[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:37 dist-test-slave-3qdh krb5kdc[19600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:37 dist-test-slave-3qdh krb5kdc[19600](info): set up 2 sockets
Jun 19 14:51:37 dist-test-slave-3qdh krb5kdc[19600](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 14:51:37.999706 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:51:38 dist-test-slave-3qdh krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:51:38.109020 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:42751
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:42751
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:38.205368 19613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:38.205569 19613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:38.205603 19613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:38.209211 19613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:38.209259 19613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:38.209276 19613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:38.209290 19613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:38.209304 19613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:38.213799 19613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:42751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:42751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:38.214608 19613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:38.215410 19613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:38.223358 19621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:38.223402 19619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:38.223357 19618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:38.223395 19613 server_base.cc:1034] running on GCE node
I20260619 14:51:38.224114 19613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:38.225188 19613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:38.226370 19613 hybrid_clock.cc:648] HybridClock initialized: now 1781880698226355 us; error 31 us; skew 500 ppm
Jun 19 14:51:38 dist-test-slave-3qdh krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880698, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:38.228964 19613 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:51:38.229781 19613 webserver.cc:460] Webserver started at http://127.19.34.126:34393/ using document root <none> and password file <none>
I20260619 14:51:38.230293 19613 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:38.230393 19613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:38.230603 19613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:38.232424 19613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "3175c4d81b1e455e85d0e711bcf86942"
format_stamp: "Formatted at 2026-06-19 14:51:38 on dist-test-slave-3qdh"
server_key: "7ade1d68210836b08324cee9f322ccf0"
server_key_iv: "436cf9b33ec1b093cbc32319f9ac87be"
server_key_version: "encryptionkey@0"
I20260619 14:51:38.233016 19613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "3175c4d81b1e455e85d0e711bcf86942"
format_stamp: "Formatted at 2026-06-19 14:51:38 on dist-test-slave-3qdh"
server_key: "7ade1d68210836b08324cee9f322ccf0"
server_key_iv: "436cf9b33ec1b093cbc32319f9ac87be"
server_key_version: "encryptionkey@0"
I20260619 14:51:38.236364 19613 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:51:38.238423 19628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:38.239552 19613 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:38.239703 19613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "3175c4d81b1e455e85d0e711bcf86942"
format_stamp: "Formatted at 2026-06-19 14:51:38 on dist-test-slave-3qdh"
server_key: "7ade1d68210836b08324cee9f322ccf0"
server_key_iv: "436cf9b33ec1b093cbc32319f9ac87be"
server_key_version: "encryptionkey@0"
I20260619 14:51:38.239809 19613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:38.258360 19613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:38.260897 19613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:38.261107 19613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:38.267392 19613 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:42751
I20260619 14:51:38.267402 19680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:42751 every 8 connection(s)
I20260619 14:51:38.268183 19613 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:38.270594 19681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:38.273808 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 19613
I20260619 14:51:38.273957 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:51:38.275008 19593 external_mini_cluster.cc:1428] Setting key 50f437420b221c9aa90ee4c3d908e6da
I20260619 14:51:38.275295 19681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942: Bootstrap starting.
I20260619 14:51:38.277514 19681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:38.278258 19681 log.cc:824] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:38.279846 19681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942: No bootstrap required, opened a new log
I20260619 14:51:38.283224 19681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3175c4d81b1e455e85d0e711bcf86942" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42751 } }
I20260619 14:51:38.283448 19681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:38.283501 19681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3175c4d81b1e455e85d0e711bcf86942, State: Initialized, Role: FOLLOWER
I20260619 14:51:38.283914 19681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3175c4d81b1e455e85d0e711bcf86942" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42751 } }
I20260619 14:51:38.284049 19681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:38.284123 19681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:38.284225 19681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:38.284922 19681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3175c4d81b1e455e85d0e711bcf86942" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42751 } }
I20260619 14:51:38.285199 19681 leader_election.cc:302] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3175c4d81b1e455e85d0e711bcf86942; no voters:
I20260619 14:51:38.285470 19681 leader_election.cc:288] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:38.285521 19686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:38.285691 19686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942 [term 1 LEADER]: Becoming Leader. State: Replica: 3175c4d81b1e455e85d0e711bcf86942, State: Running, Role: LEADER
I20260619 14:51:38.285938 19686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3175c4d81b1e455e85d0e711bcf86942" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42751 } }
I20260619 14:51:38.286492 19681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:38.287374 19688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3175c4d81b1e455e85d0e711bcf86942. Latest consensus state: current_term: 1 leader_uuid: "3175c4d81b1e455e85d0e711bcf86942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3175c4d81b1e455e85d0e711bcf86942" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42751 } } }
I20260619 14:51:38.287384 19687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3175c4d81b1e455e85d0e711bcf86942" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3175c4d81b1e455e85d0e711bcf86942" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42751 } } }
I20260619 14:51:38.287572 19687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:38.287819 19688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:38.288123 19695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:38.290974 19695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 19 14:51:38 dist-test-slave-3qdh krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:38.296006 19695 catalog_manager.cc:1347] Generated new cluster ID: eb12ce46f64843f78661fa77732ee8b9
I20260619 14:51:38.296060 19695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:38.296932 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:38.277481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.19.34.126:42751)
0619 14:51:38.277853 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:38.277867 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:38.279068 (+ 1201us) server_negotiation.cc:407] Connection header received
0619 14:51:38.282920 (+ 3852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:38.282938 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:38.283216 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:38.283539 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:38.285669 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:38.286743 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:38.288375 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:38.288986 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:38.289120 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:38.292203 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:38.292207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:38.292216 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:38.292242 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:38.293904 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:38.294431 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:38.294438 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:38.294442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:38.294563 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:38.294916 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:38.294919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:38.294920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:38.295273 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:38.295473 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:38.295986 (+ 513us) server_negotiation.cc:299] Negotiation successful
0619 14:51:38.296207 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":155,"threads_started":1}
I20260619 14:51:38.305024 19695 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:38.305964 19695 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:38.315932 19695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3175c4d81b1e455e85d0e711bcf86942: Generated new TSK 0
I20260619 14:51:38.316583 19695 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.34.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
I20260619 14:51:38.341447 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:42751
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:38.441394 19707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:38.441605 19707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:38.441648 19707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:38.445693 19707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:38.445740 19707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:38.445847 19707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:51:38.450685 19707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:42751
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:38.451546 19707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:38.452471 19707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:51:38.461002 19707 server_base.cc:1034] running on GCE node
W20260619 14:51:38.460937 19715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:38.460937 19712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:38.460943 19713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:38.461575 19707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:38.462121 19707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:38.463308 19707 hybrid_clock.cc:648] HybridClock initialized: now 1781880698463297 us; error 27 us; skew 500 ppm
Jun 19 14:51:38 dist-test-slave-3qdh krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880698, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:38.465891 19707 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:51:38.466643 19707 webserver.cc:460] Webserver started at http://127.19.34.65:40913/ using document root <none> and password file <none>
I20260619 14:51:38.467067 19707 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:38.467115 19707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:38.467262 19707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:38.469120 19707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "10ce2ace11734e518350200669e81681"
format_stamp: "Formatted at 2026-06-19 14:51:38 on dist-test-slave-3qdh"
server_key: "9b6bf0dcc57482c98f5f9ee298cfe39c"
server_key_iv: "516116d26786a645e0f85936f974ab40"
server_key_version: "encryptionkey@0"
I20260619 14:51:38.469588 19707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "10ce2ace11734e518350200669e81681"
format_stamp: "Formatted at 2026-06-19 14:51:38 on dist-test-slave-3qdh"
server_key: "9b6bf0dcc57482c98f5f9ee298cfe39c"
server_key_iv: "516116d26786a645e0f85936f974ab40"
server_key_version: "encryptionkey@0"
I20260619 14:51:38.472929 19707 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:51:38.474985 19722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:38.476109 19707 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:38.476244 19707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "10ce2ace11734e518350200669e81681"
format_stamp: "Formatted at 2026-06-19 14:51:38 on dist-test-slave-3qdh"
server_key: "9b6bf0dcc57482c98f5f9ee298cfe39c"
server_key_iv: "516116d26786a645e0f85936f974ab40"
server_key_version: "encryptionkey@0"
I20260619 14:51:38.476380 19707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:38.500476 19707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:38.503489 19707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:38.503681 19707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:38.504216 19707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:38.504979 19707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:38.505050 19707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:38.505125 19707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:38.505182 19707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:38.513147 19707 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:42721
I20260619 14:51:38.513180 19835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:42721 every 8 connection(s)
I20260619 14:51:38.513943 19707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:51:38.516660 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 19707
I20260619 14:51:38.516786 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:51:38.517030 19593 external_mini_cluster.cc:1428] Setting key b141daf6ef5ea8e3a575b4c8b2e5c9b6
Jun 19 14:51:38 dist-test-slave-3qdh krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880698, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.34.66@KRBTEST.COM; defaulting to no policy
I20260619 14:51:38.528175 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:38.515551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:48145 (local address 127.19.34.126:42751)
0619 14:51:38.515700 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:38.515705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:38.516852 (+ 1147us) server_negotiation.cc:407] Connection header received
0619 14:51:38.517947 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:38.517950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:38.517991 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:38.518060 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:38.519237 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:38.520088 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:38.522004 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:38.522357 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:38.522490 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:38.524821 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:38.524832 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:38.524834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:38.524856 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:38.526304 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:38.526730 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:38.526733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:38.526734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:38.526804 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:38.527146 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:38.527149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:38.527151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:38.527340 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:38.527440 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:38.527862 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 14:51:38.528003 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.66@KRBTEST.COM" created.
I20260619 14:51:38.528926 19838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:38.515946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:42751 (local address 127.19.34.65:48145)
0619 14:51:38.516386 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:38.516423 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:38.517750 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:38.518210 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:38.518217 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:38.518629 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:38.519084 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:38.519097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:38.520260 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:38.520284 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:38.521822 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:38.521832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:38.522520 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:38.522524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:38.522693 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:38.523241 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:38.523257 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:38.524648 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:38.526401 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:38.526406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:38.526417 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:38.526635 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:38.526883 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:38.526887 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:38.526889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:38.527054 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:38.527440 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:38.527446 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:38.527647 (+ 201us) client_negotiation.cc:769] Sending connection context
0619 14:51:38.527808 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 14:51:38.528109 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":101,"threads_started":1}
I20260619 14:51:38.530155 19836 heartbeater.cc:347] Connected to a master server at 127.19.34.126:42751
I20260619 14:51:38.530386 19836 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:38.530826 19836 heartbeater.cc:511] Master 127.19.34.126:42751 requested a full tablet report, sending...
I20260619 14:51:38.532090 19645 ts_manager.cc:194] Registered new tserver with Master: 10ce2ace11734e518350200669e81681 (127.19.34.65:42721)
I20260619 14:51:38.533180 19645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:48145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
I20260619 14:51:38.543411 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:42751
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:38.643739 19841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:38.643981 19841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:38.644019 19841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:38.647686 19841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:38.647734 19841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:38.647794 19841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:51:38.652180 19841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:42751
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:38.653131 19841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:38.654019 19841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:38.662551 19846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:38.662609 19849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:38.662711 19847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:38.662747 19841 server_base.cc:1034] running on GCE node
I20260619 14:51:38.663175 19841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:38.663761 19841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:38.664937 19841 hybrid_clock.cc:648] HybridClock initialized: now 1781880698664920 us; error 31 us; skew 500 ppm
Jun 19 14:51:38 dist-test-slave-3qdh krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880698, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:38.667181 19841 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:51:38.667948 19841 webserver.cc:460] Webserver started at http://127.19.34.66:34877/ using document root <none> and password file <none>
I20260619 14:51:38.668474 19841 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:38.668529 19841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:38.668725 19841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:38.670543 19841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "b160cf33bbea432a8507808a7a47d8da"
format_stamp: "Formatted at 2026-06-19 14:51:38 on dist-test-slave-3qdh"
server_key: "a562cf05b3af3f1b0ef6ee9be96624a2"
server_key_iv: "79a75808c935941455b3e46f68bcb520"
server_key_version: "encryptionkey@0"
I20260619 14:51:38.671036 19841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "b160cf33bbea432a8507808a7a47d8da"
format_stamp: "Formatted at 2026-06-19 14:51:38 on dist-test-slave-3qdh"
server_key: "a562cf05b3af3f1b0ef6ee9be96624a2"
server_key_iv: "79a75808c935941455b3e46f68bcb520"
server_key_version: "encryptionkey@0"
I20260619 14:51:38.674347 19841 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:51:38.676515 19856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:38.677510 19841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:38.677623 19841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "b160cf33bbea432a8507808a7a47d8da"
format_stamp: "Formatted at 2026-06-19 14:51:38 on dist-test-slave-3qdh"
server_key: "a562cf05b3af3f1b0ef6ee9be96624a2"
server_key_iv: "79a75808c935941455b3e46f68bcb520"
server_key_version: "encryptionkey@0"
I20260619 14:51:38.677714 19841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:38.687225 19841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:38.690186 19841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:38.690363 19841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:38.690837 19841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:38.691542 19841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:38.691586 19841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:38.691648 19841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:38.691686 19841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:38.699955 19841 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:35215
I20260619 14:51:38.699975 19969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:35215 every 8 connection(s)
I20260619 14:51:38.700778 19841 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:51:38.708153 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 19841
I20260619 14:51:38.708273 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:51:38.708582 19593 external_mini_cluster.cc:1428] Setting key 8f48e52f9985153124dcc4b1c34c0e88
Jun 19 14:51:38 dist-test-slave-3qdh krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880698, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:38.714087 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:38.702410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:45507 (local address 127.19.34.126:42751)
0619 14:51:38.702576 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:38.702579 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:38.703771 (+ 1192us) server_negotiation.cc:407] Connection header received
0619 14:51:38.704555 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:38.704559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:38.704609 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:38.704701 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:38.705739 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:38.706267 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:38.707309 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:38.707518 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:38.707626 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:38.710345 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:38.710349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:38.710352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:38.710378 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:38.711940 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:38.712607 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:38.712609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:38.712611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:38.712686 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:38.713070 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:38.713072 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:38.713073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:38.713250 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:38.713323 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:38.713898 (+ 575us) server_negotiation.cc:299] Negotiation successful
0619 14:51:38.713982 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 14:51:38.715140 19972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:38.702754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:42751 (local address 127.19.34.66:45507)
0619 14:51:38.703600 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:38.703638 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:38.704376 (+ 738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:38.704854 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:38.704859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:38.705135 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:38.705588 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:38.705600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:38.706395 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:38.706399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:38.707187 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:38.707196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:38.707674 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:38.707678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:38.707885 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:38.708502 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:38.708526 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:38.710186 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:38.712065 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:38.712072 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:38.712086 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:38.712504 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:38.712779 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:38.712784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:38.712789 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:38.712966 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:38.713352 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:38.713360 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:38.713648 (+ 288us) client_negotiation.cc:769] Sending connection context
0619 14:51:38.713851 (+ 203us) client_negotiation.cc:240] Negotiation successful
0619 14:51:38.714169 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":551,"threads_started":1}
I20260619 14:51:38.716245 19970 heartbeater.cc:347] Connected to a master server at 127.19.34.126:42751
I20260619 14:51:38.716529 19970 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:38.716958 19970 heartbeater.cc:511] Master 127.19.34.126:42751 requested a full tablet report, sending...
I20260619 14:51:38.717907 19645 ts_manager.cc:194] Registered new tserver with Master: b160cf33bbea432a8507808a7a47d8da (127.19.34.66:35215)
I20260619 14:51:38.718425 19645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:45507
WARNING: no policy specified for kudu/127.19.34.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
I20260619 14:51:38.734586 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:42751
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:38.839344 19975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:38.839537 19975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:38.839574 19975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:38.843276 19975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:38.843323 19975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:38.843426 19975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:51:38.848066 19975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:42751
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:38.848930 19975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:38.849754 19975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:38.858166 19983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:38.858263 19980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:38.858376 19975 server_base.cc:1034] running on GCE node
W20260619 14:51:38.858464 19981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:38.858933 19975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:38.859462 19975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:38.860596 19975 hybrid_clock.cc:648] HybridClock initialized: now 1781880698860568 us; error 39 us; skew 500 ppm
Jun 19 14:51:38 dist-test-slave-3qdh krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880698, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:38.863066 19975 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:51:38.863875 19975 webserver.cc:460] Webserver started at http://127.19.34.67:40503/ using document root <none> and password file <none>
I20260619 14:51:38.864406 19975 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:38.864460 19975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:38.864646 19975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:38.866305 19975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "ecdb4e70a1e84e12bb38ef13da2a7047"
format_stamp: "Formatted at 2026-06-19 14:51:38 on dist-test-slave-3qdh"
server_key: "363cf2edf6c8a82ab1c130ac2f6d6d50"
server_key_iv: "adb839dfd0420ff79138af3246c594b2"
server_key_version: "encryptionkey@0"
I20260619 14:51:38.866817 19975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "ecdb4e70a1e84e12bb38ef13da2a7047"
format_stamp: "Formatted at 2026-06-19 14:51:38 on dist-test-slave-3qdh"
server_key: "363cf2edf6c8a82ab1c130ac2f6d6d50"
server_key_iv: "adb839dfd0420ff79138af3246c594b2"
server_key_version: "encryptionkey@0"
I20260619 14:51:38.869973 19975 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:51:38.871866 19990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:38.872990 19975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:38.873118 19975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "ecdb4e70a1e84e12bb38ef13da2a7047"
format_stamp: "Formatted at 2026-06-19 14:51:38 on dist-test-slave-3qdh"
server_key: "363cf2edf6c8a82ab1c130ac2f6d6d50"
server_key_iv: "adb839dfd0420ff79138af3246c594b2"
server_key_version: "encryptionkey@0"
I20260619 14:51:38.873217 19975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:38.894918 19975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:38.897521 19975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:38.897687 19975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:38.898147 19975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:38.898795 19975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:38.898855 19975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:38.898907 19975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:38.898944 19975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:38.907030 19975 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:34131
I20260619 14:51:38.907055 20103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:34131 every 8 connection(s)
I20260619 14:51:38.907765 19975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:51:38.909348 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 19975
I20260619 14:51:38.909459 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:51:38.909695 19593 external_mini_cluster.cc:1428] Setting key 1c16d8c7dce282009beb1a860547477a
Jun 19 14:51:38 dist-test-slave-3qdh krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880698, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:38.919260 19683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:38.909414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:50741 (local address 127.19.34.126:42751)
0619 14:51:38.909542 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:38.909546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:38.910282 (+ 736us) server_negotiation.cc:407] Connection header received
0619 14:51:38.911377 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:38.911380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:38.911426 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:38.911493 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:38.912451 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:38.912976 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:38.914080 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:38.914287 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:38.914386 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:38.916360 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:38.916362 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:38.916364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:38.916384 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:38.917656 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:38.918073 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:38.918075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:38.918076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:38.918139 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:38.918435 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:38.918437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:38.918443 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:38.918595 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:38.918666 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:38.919053 (+ 387us) server_negotiation.cc:299] Negotiation successful
0619 14:51:38.919136 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:51:38.919910 20106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:38.909721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:42751 (local address 127.19.34.67:50741)
0619 14:51:38.910165 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:38.910207 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:38.911205 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:38.911636 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:38.911641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:38.911921 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:38.912285 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:38.912339 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:38.913097 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:38.913100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:38.913965 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:38.913973 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:38.914424 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:38.914428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:38.914577 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:38.915006 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:38.915019 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:38.916187 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:38.917747 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:38.917751 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:38.917760 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:38.917990 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:38.918224 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:38.918241 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:38.918242 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:38.918345 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:38.918676 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:38.918680 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:38.918868 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 14:51:38.919007 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 14:51:38.919209 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":120,"threads_started":1}
I20260619 14:51:38.920841 20104 heartbeater.cc:347] Connected to a master server at 127.19.34.126:42751
I20260619 14:51:38.921001 20104 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:38.921361 20104 heartbeater.cc:511] Master 127.19.34.126:42751 requested a full tablet report, sending...
I20260619 14:51:38.922181 19645 ts_manager.cc:194] Registered new tserver with Master: ecdb4e70a1e84e12bb38ef13da2a7047 (127.19.34.67:34131)
I20260619 14:51:38.922741 19645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:50741
I20260619 14:51:38.923354 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:51:38 dist-test-slave-3qdh krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880698, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:51:38 dist-test-slave-3qdh krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880698, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.34.65@KRBTEST.COM
I20260619 14:51:38.946550 20111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:38.938241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35122 (local address 127.19.34.65:42721)
0619 14:51:38.938524 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:38.938535 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:38.938555 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:51:38.938596 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:38.938600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:38.938758 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:38.938863 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:38.939445 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:38.940177 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:38.940938 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:38.941267 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:38.941346 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:38.943488 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:38.943496 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:38.943506 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:38.943528 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:38.945052 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:38.945402 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:38.945406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:38.945407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:38.945476 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:38.945704 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:38.945706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:38.945707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:38.945905 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:38.946046 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:38.946201 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 14:51:38.946375 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":96,"threads_started":1}
W20260619 14:51:38.947386 19750 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35122
Jun 19 14:51:38 dist-test-slave-3qdh krb5kdc[19600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:51:38 dist-test-slave-3qdh krb5kdc[19600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.65@KRBTEST.COM
I20260619 14:51:38.969789 20111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:38.962112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35138 (local address 127.19.34.65:42721)
0619 14:51:38.962252 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:38.962257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:38.962387 (+ 130us) server_negotiation.cc:407] Connection header received
0619 14:51:38.962553 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:38.962557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:38.962605 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:38.962687 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:38.963322 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:38.963834 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:38.964656 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:38.964887 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:38.964998 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:38.967034 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:38.967041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:38.967042 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:38.967061 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:38.968088 (+ 1027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:38.968580 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:38.968585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:38.968587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:38.968664 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:38.968937 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:38.968940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:38.968942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:38.969094 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:38.969228 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:38.969458 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 14:51:38.969582 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:51:38.970811 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 19707
I20260619 14:51:38.976051 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 19841
I20260619 14:51:38.980894 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 19975
I20260619 14:51:38.985491 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 19613
2026-06-19T14:51:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1964 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:39 dist-test-slave-3qdh krb5kdc[20118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:39 dist-test-slave-3qdh krb5kdc[20118](info): set up 2 sockets
Jun 19 14:51:39 dist-test-slave-3qdh krb5kdc[20118](info): commencing operation
krb5kdc: starting...
W20260619 14:51:39.971896 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:51:40 dist-test-slave-3qdh krb5kdc[20118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:51:40.082852 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:38093
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:42195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:38093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:40.181639 20131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:40.181885 20131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:40.181948 20131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:40.185729 20131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:40.185796 20131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:40.185830 20131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:40.185879 20131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:40.185910 20131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:40.190783 20131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:42195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:38093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:38093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:40.191716 20131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:40.192725 20131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:40.200147 20139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:40.200233 20137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:40.200333 20131 server_base.cc:1034] running on GCE node
W20260619 14:51:40.200344 20136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:40.200971 20131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:40.201889 20131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:40.203071 20131 hybrid_clock.cc:648] HybridClock initialized: now 1781880700203047 us; error 39 us; skew 500 ppm
Jun 19 14:51:40 dist-test-slave-3qdh krb5kdc[20118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880700, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:40.205629 20131 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:51:40.206367 20131 webserver.cc:460] Webserver started at http://127.19.34.126:35257/ using document root <none> and password file <none>
I20260619 14:51:40.206895 20131 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:40.206985 20131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:40.207198 20131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:40.209056 20131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "dd1d49e2f1c24852b2181764f966826a"
format_stamp: "Formatted at 2026-06-19 14:51:40 on dist-test-slave-3qdh"
server_key: "3d2127a5200c2c8adecfd804d0f0f406"
server_key_iv: "ff0ea77ea5f539095ce4a14d33c000f4"
server_key_version: "encryptionkey@0"
I20260619 14:51:40.209585 20131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "dd1d49e2f1c24852b2181764f966826a"
format_stamp: "Formatted at 2026-06-19 14:51:40 on dist-test-slave-3qdh"
server_key: "3d2127a5200c2c8adecfd804d0f0f406"
server_key_iv: "ff0ea77ea5f539095ce4a14d33c000f4"
server_key_version: "encryptionkey@0"
I20260619 14:51:40.213068 20131 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 14:51:40.215178 20146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:40.216261 20131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:51:40.216455 20131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "dd1d49e2f1c24852b2181764f966826a"
format_stamp: "Formatted at 2026-06-19 14:51:40 on dist-test-slave-3qdh"
server_key: "3d2127a5200c2c8adecfd804d0f0f406"
server_key_iv: "ff0ea77ea5f539095ce4a14d33c000f4"
server_key_version: "encryptionkey@0"
I20260619 14:51:40.216567 20131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:40.238493 20131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:40.240965 20131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:40.241151 20131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:40.247810 20131 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:38093
I20260619 14:51:40.247848 20198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:38093 every 8 connection(s)
I20260619 14:51:40.248713 20131 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:40.250983 20199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:40.255872 20199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a: Bootstrap starting.
I20260619 14:51:40.257812 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 20131
I20260619 14:51:40.257907 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:51:40.258064 20199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:40.258201 19593 external_mini_cluster.cc:1428] Setting key 170b0d8f0a2606a0f4e5f22efadade2c
I20260619 14:51:40.258855 20199 log.cc:824] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:40.260833 20199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a: No bootstrap required, opened a new log
I20260619 14:51:40.264822 20199 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1d49e2f1c24852b2181764f966826a" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38093 } }
I20260619 14:51:40.265071 20199 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:40.265111 20199 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd1d49e2f1c24852b2181764f966826a, State: Initialized, Role: FOLLOWER
I20260619 14:51:40.265524 20199 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1d49e2f1c24852b2181764f966826a" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38093 } }
I20260619 14:51:40.265646 20199 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:40.265738 20199 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:40.265846 20199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:40.266657 20199 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1d49e2f1c24852b2181764f966826a" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38093 } }
Jun 19 14:51:40 dist-test-slave-3qdh krb5kdc[20118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:40.266965 20199 leader_election.cc:302] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd1d49e2f1c24852b2181764f966826a; no voters:
I20260619 14:51:40.267238 20199 leader_election.cc:288] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:40.267342 20204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:40.267553 20204 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a [term 1 LEADER]: Becoming Leader. State: Replica: dd1d49e2f1c24852b2181764f966826a, State: Running, Role: LEADER
I20260619 14:51:40.267859 20204 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1d49e2f1c24852b2181764f966826a" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38093 } }
I20260619 14:51:40.268494 20199 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:40.269929 20206 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd1d49e2f1c24852b2181764f966826a. Latest consensus state: current_term: 1 leader_uuid: "dd1d49e2f1c24852b2181764f966826a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1d49e2f1c24852b2181764f966826a" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38093 } } }
I20260619 14:51:40.269929 20205 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd1d49e2f1c24852b2181764f966826a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1d49e2f1c24852b2181764f966826a" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38093 } } }
I20260619 14:51:40.270078 20206 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:40.270097 20205 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:40.270510 20217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:40.273063 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:40.259304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42038 (local address 127.19.34.126:38093)
0619 14:51:40.259741 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:40.259759 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:40.259793 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:51:40.260496 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:40.260515 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:40.260827 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:40.261143 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:40.261777 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.263165 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.264015 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.264739 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.264900 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:40.267707 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:40.267713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:40.267728 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:40.267755 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:40.269743 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:40.270251 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:40.270259 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:40.270265 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:40.270384 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:40.270718 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:40.270722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:40.270724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:40.271179 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:40.271364 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:40.271993 (+ 629us) server_negotiation.cc:299] Negotiation successful
0619 14:51:40.272239 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":124,"threads_started":1}
I20260619 14:51:40.273737 20217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:40.277979 20217 catalog_manager.cc:1347] Generated new cluster ID: 86818128b141464ab3c8349577631df3
I20260619 14:51:40.278033 20217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:40.301990 20217 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:40.302824 20217 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:40.310053 20217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd1d49e2f1c24852b2181764f966826a: Generated new TSK 0
I20260619 14:51:40.310627 20217 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.34.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
I20260619 14:51:40.338366 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:38093
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:42195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:40.462726 20225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:40.462983 20225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:40.463024 20225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:40.466627 20225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:40.466670 20225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:40.466717 20225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:51:40.471472 20225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:42195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:38093
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:40.472218 20225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:40.473151 20225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:40.481465 20233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:40.481479 20225 server_base.cc:1034] running on GCE node
W20260619 14:51:40.481568 20231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:40.481570 20230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:40.481998 20225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:40.482509 20225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:40.483655 20225 hybrid_clock.cc:648] HybridClock initialized: now 1781880700483629 us; error 40 us; skew 500 ppm
Jun 19 14:51:40 dist-test-slave-3qdh krb5kdc[20118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880700, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:40.486050 20225 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:51:40.486828 20225 webserver.cc:460] Webserver started at http://127.19.34.65:37247/ using document root <none> and password file <none>
I20260619 14:51:40.487339 20225 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:40.487417 20225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:40.487633 20225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:40.489569 20225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "6bc1d2eea1d84b6b8e7e3dd5c3e6e68c"
format_stamp: "Formatted at 2026-06-19 14:51:40 on dist-test-slave-3qdh"
server_key: "d45f07cdda93253684d7a7ab5af429a7"
server_key_iv: "36a7e23cffaccef78527ac3cafdcc887"
server_key_version: "encryptionkey@0"
I20260619 14:51:40.490069 20225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "6bc1d2eea1d84b6b8e7e3dd5c3e6e68c"
format_stamp: "Formatted at 2026-06-19 14:51:40 on dist-test-slave-3qdh"
server_key: "d45f07cdda93253684d7a7ab5af429a7"
server_key_iv: "36a7e23cffaccef78527ac3cafdcc887"
server_key_version: "encryptionkey@0"
I20260619 14:51:40.493432 20225 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 14:51:40.495491 20240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:40.496577 20225 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:40.496694 20225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "6bc1d2eea1d84b6b8e7e3dd5c3e6e68c"
format_stamp: "Formatted at 2026-06-19 14:51:40 on dist-test-slave-3qdh"
server_key: "d45f07cdda93253684d7a7ab5af429a7"
server_key_iv: "36a7e23cffaccef78527ac3cafdcc887"
server_key_version: "encryptionkey@0"
I20260619 14:51:40.496794 20225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:40.514834 20225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:40.517109 20225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:40.517292 20225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:40.517757 20225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:40.518494 20225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:40.518563 20225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:40.518640 20225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:40.518685 20225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:40.526894 20225 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:40305
I20260619 14:51:40.526922 20353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:40305 every 8 connection(s)
I20260619 14:51:40.527695 20225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:51:40.533546 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 20225
I20260619 14:51:40.533681 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:51:40.533943 19593 external_mini_cluster.cc:1428] Setting key fe752de7f0b90f1caefd8d8170de038d
Jun 19 14:51:40 dist-test-slave-3qdh krb5kdc[20118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880700, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:40.540535 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:40.529331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:45127 (local address 127.19.34.126:38093)
0619 14:51:40.529449 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:40.529458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:40.530138 (+ 680us) server_negotiation.cc:407] Connection header received
0619 14:51:40.531205 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:40.531207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:40.531252 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:40.531318 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:40.532275 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.532852 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.534016 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.534334 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.534439 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:40.537112 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:40.537117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:40.537121 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:40.537150 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:40.538668 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:40.539139 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:40.539144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:40.539149 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:40.539246 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:40.539611 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:40.539617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:40.539619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:40.539823 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:40.539923 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:40.540337 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 14:51:40.540435 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:51:40.541217 20356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:40.529599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:38093 (local address 127.19.34.65:45127)
0619 14:51:40.529975 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:40.530011 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:40.531032 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:40.531454 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:40.531459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:40.531758 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:40.532140 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:40.532151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.532952 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.532955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:40.533901 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:40.533909 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.534477 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.534480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:40.534599 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:40.535133 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:40.535150 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:40.536928 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:40.538776 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:40.538780 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:40.538806 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:40.539042 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:40.539369 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:40.539374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:40.539375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:40.539518 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:40.539919 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:40.539923 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:40.540115 (+ 192us) client_negotiation.cc:769] Sending connection context
0619 14:51:40.540248 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 14:51:40.540492 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":99,"threads_started":1}
I20260619 14:51:40.542325 20354 heartbeater.cc:347] Connected to a master server at 127.19.34.126:38093
I20260619 14:51:40.542544 20354 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:40.542961 20354 heartbeater.cc:511] Master 127.19.34.126:38093 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.34.66@KRBTEST.COM; defaulting to no policy
I20260619 14:51:40.544466 20163 ts_manager.cc:194] Registered new tserver with Master: 6bc1d2eea1d84b6b8e7e3dd5c3e6e68c (127.19.34.65:40305)
I20260619 14:51:40.545424 20163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:45127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
I20260619 14:51:40.558326 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:38093
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:42195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:40.657251 20359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:40.657490 20359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:40.657564 20359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:40.661245 20359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:40.661339 20359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:40.661414 20359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:51:40.666237 20359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:42195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:38093
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:40.667110 20359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:40.667969 20359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:40.676793 20367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:40.676863 20365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:40.676846 20364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:40.676839 20359 server_base.cc:1034] running on GCE node
I20260619 14:51:40.677342 20359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:40.677863 20359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:40.679064 20359 hybrid_clock.cc:648] HybridClock initialized: now 1781880700679038 us; error 39 us; skew 500 ppm
Jun 19 14:51:40 dist-test-slave-3qdh krb5kdc[20118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880700, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:40.681574 20359 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:51:40.682292 20359 webserver.cc:460] Webserver started at http://127.19.34.66:37627/ using document root <none> and password file <none>
I20260619 14:51:40.682796 20359 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:40.682878 20359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:40.683086 20359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:40.685012 20359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "0ac295d70b6e48eaaaf580e3ab6ccde1"
format_stamp: "Formatted at 2026-06-19 14:51:40 on dist-test-slave-3qdh"
server_key: "1a5a5a7b3e358ab91a2f9428325fa027"
server_key_iv: "1ac24d9ba048de2cd00f8b55aea6968d"
server_key_version: "encryptionkey@0"
I20260619 14:51:40.685498 20359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "0ac295d70b6e48eaaaf580e3ab6ccde1"
format_stamp: "Formatted at 2026-06-19 14:51:40 on dist-test-slave-3qdh"
server_key: "1a5a5a7b3e358ab91a2f9428325fa027"
server_key_iv: "1ac24d9ba048de2cd00f8b55aea6968d"
server_key_version: "encryptionkey@0"
I20260619 14:51:40.688784 20359 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:51:40.690743 20374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:40.691783 20359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:40.691915 20359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "0ac295d70b6e48eaaaf580e3ab6ccde1"
format_stamp: "Formatted at 2026-06-19 14:51:40 on dist-test-slave-3qdh"
server_key: "1a5a5a7b3e358ab91a2f9428325fa027"
server_key_iv: "1ac24d9ba048de2cd00f8b55aea6968d"
server_key_version: "encryptionkey@0"
I20260619 14:51:40.692006 20359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:40.707607 20359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:40.710080 20359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:40.710256 20359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:40.710759 20359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:40.711431 20359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:40.711493 20359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:40.711549 20359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:40.711596 20359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:40.719326 20359 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:36127
I20260619 14:51:40.719383 20487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:36127 every 8 connection(s)
I20260619 14:51:40.720108 20359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:51:40.722905 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 20359
I20260619 14:51:40.723037 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:51:40.723284 19593 external_mini_cluster.cc:1428] Setting key 30707051141fa0933005be0218758a0d
Jun 19 14:51:40 dist-test-slave-3qdh krb5kdc[20118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880700, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.34.67@KRBTEST.COM; defaulting to no policy
I20260619 14:51:40.734002 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:40.721722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:53691 (local address 127.19.34.126:38093)
0619 14:51:40.721862 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:40.721867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:40.722526 (+ 659us) server_negotiation.cc:407] Connection header received
0619 14:51:40.723861 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:40.723864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:40.723926 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:40.724022 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:40.725233 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.726013 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.727719 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.727959 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.728034 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:40.730663 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:40.730665 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:40.730668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:40.730689 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:40.731979 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:40.732530 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:40.732533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:40.732536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:40.732635 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:40.733075 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:40.733077 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:40.733079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:40.733239 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:40.733312 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:40.733796 (+ 484us) server_negotiation.cc:299] Negotiation successful
0619 14:51:40.733890 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:51:40.734802 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:40.722029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:38093 (local address 127.19.34.66:53691)
0619 14:51:40.722396 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:40.722432 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:40.723662 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:40.724191 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:40.724200 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:40.724653 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:40.725094 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:40.725105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.726146 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.726151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:40.727594 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:40.727608 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.728101 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.728107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:40.728341 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:40.728971 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:40.728992 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:40.730503 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:40.732086 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:40.732090 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:40.732102 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:40.732382 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:40.732774 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:40.732779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:40.732781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:40.732972 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:40.733327 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:40.733331 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:40.733550 (+ 219us) client_negotiation.cc:769] Sending connection context
0619 14:51:40.733726 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 14:51:40.733945 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":88,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.67@KRBTEST.COM" created.
I20260619 14:51:40.735965 20488 heartbeater.cc:347] Connected to a master server at 127.19.34.126:38093
I20260619 14:51:40.736202 20488 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:40.736630 20488 heartbeater.cc:511] Master 127.19.34.126:38093 requested a full tablet report, sending...
I20260619 14:51:40.737623 20163 ts_manager.cc:194] Registered new tserver with Master: 0ac295d70b6e48eaaaf580e3ab6ccde1 (127.19.34.66:36127)
I20260619 14:51:40.738449 20163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:53691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
I20260619 14:51:40.750532 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:38093
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:42195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:40.851346 20493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:40.851533 20493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:40.851567 20493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:40.855250 20493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:40.855320 20493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:40.855423 20493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:51:40.860512 20493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:42195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:38093
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:40.861367 20493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:40.862221 20493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:40.870965 20501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:40.871026 20498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:40.871049 20499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:40.871160 20493 server_base.cc:1034] running on GCE node
I20260619 14:51:40.871518 20493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:40.872000 20493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:40.873165 20493 hybrid_clock.cc:648] HybridClock initialized: now 1781880700873149 us; error 35 us; skew 500 ppm
Jun 19 14:51:40 dist-test-slave-3qdh krb5kdc[20118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880700, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:40.875571 20493 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:51:40.876381 20493 webserver.cc:460] Webserver started at http://127.19.34.67:37553/ using document root <none> and password file <none>
I20260619 14:51:40.876832 20493 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:40.876880 20493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:40.877005 20493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:40.878618 20493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "5835309538e74a7a8116d3fe430e50ed"
format_stamp: "Formatted at 2026-06-19 14:51:40 on dist-test-slave-3qdh"
server_key: "478ef439171f5172cceebe487973472a"
server_key_iv: "16c02d9e00cabafb81203e00b25321ce"
server_key_version: "encryptionkey@0"
I20260619 14:51:40.879067 20493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "5835309538e74a7a8116d3fe430e50ed"
format_stamp: "Formatted at 2026-06-19 14:51:40 on dist-test-slave-3qdh"
server_key: "478ef439171f5172cceebe487973472a"
server_key_iv: "16c02d9e00cabafb81203e00b25321ce"
server_key_version: "encryptionkey@0"
I20260619 14:51:40.882104 20493 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:51:40.884058 20508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:40.885234 20493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:40.885340 20493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "5835309538e74a7a8116d3fe430e50ed"
format_stamp: "Formatted at 2026-06-19 14:51:40 on dist-test-slave-3qdh"
server_key: "478ef439171f5172cceebe487973472a"
server_key_iv: "16c02d9e00cabafb81203e00b25321ce"
server_key_version: "encryptionkey@0"
I20260619 14:51:40.885432 20493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:40.895510 20493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:40.897833 20493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:40.898028 20493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:40.898464 20493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:40.899132 20493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:40.899194 20493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:40.899261 20493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:40.899299 20493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:40.907573 20493 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:37517
I20260619 14:51:40.907594 20621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:37517 every 8 connection(s)
I20260619 14:51:40.908335 20493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:51:40.915390 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 20493
I20260619 14:51:40.915486 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:51:40.915709 19593 external_mini_cluster.cc:1428] Setting key 6da4de133d357b58e6c4946253596d00
Jun 19 14:51:40 dist-test-slave-3qdh krb5kdc[20118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880700, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:40.920635 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:40.909790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:34025 (local address 127.19.34.126:38093)
0619 14:51:40.909908 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:40.909911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:40.910503 (+ 592us) server_negotiation.cc:407] Connection header received
0619 14:51:40.911587 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:40.911591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:40.911648 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:40.911751 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:40.912865 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.913387 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.914483 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.914705 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.914813 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:40.917349 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:40.917352 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:40.917355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:40.917392 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:40.918743 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:40.919285 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:40.919289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:40.919291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:40.919390 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:40.919688 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:40.919692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:40.919694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:40.919859 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:40.919944 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:40.920400 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 14:51:40.920497 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:51:40.921192 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:40.910005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:38093 (local address 127.19.34.67:34025)
0619 14:51:40.910359 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:40.910392 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:40.911402 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:40.911896 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:40.911901 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:40.912237 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:40.912724 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:40.912737 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.913508 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.913512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:40.914347 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:40.914356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.914843 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.914847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:40.915005 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:40.915551 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:40.915570 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:40.917198 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:40.918869 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:40.918874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:40.918886 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:40.919176 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:40.919470 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:40.919474 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:40.919476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:40.919595 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:40.919957 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:40.919964 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:40.920185 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 14:51:40.920352 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 14:51:40.920590 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":85,"threads_started":1}
I20260619 14:51:40.922079 20622 heartbeater.cc:347] Connected to a master server at 127.19.34.126:38093
I20260619 14:51:40.922282 20622 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:40.922686 20622 heartbeater.cc:511] Master 127.19.34.126:38093 requested a full tablet report, sending...
I20260619 14:51:40.923524 20163 ts_manager.cc:194] Registered new tserver with Master: 5835309538e74a7a8116d3fe430e50ed (127.19.34.67:37517)
I20260619 14:51:40.924057 20163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:34025
I20260619 14:51:40.929239 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:51:40.938081 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:40.931999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42046 (local address 127.19.34.126:38093)
0619 14:51:40.932138 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:40.932141 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:40.932237 (+ 96us) server_negotiation.cc:407] Connection header received
0619 14:51:40.932360 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:40.932362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:40.932394 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:40.932455 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:40.933053 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.933503 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.934258 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.934450 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.934539 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:40.935518 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:40.935520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:40.935522 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:40.935540 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:40.936720 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:40.937040 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:40.937042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:40.937043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:40.937103 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:40.937301 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:40.937303 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:40.937304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:40.937475 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:40.937591 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:40.937861 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 14:51:40.937935 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:51:40.941908 20163 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:51:40.944703 20163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:51:40.957422 20636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:40.952571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51004 (local address 127.19.34.66:36127)
0619 14:51:40.952993 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:40.953006 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:40.953030 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:51:40.953111 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:40.953117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:40.953311 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:40.953782 (+ 471us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:40.953941 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.954742 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.955782 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.956637 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.956789 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:40.956897 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:40.957163 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 14:51:40.957260 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":135,"threads_started":1}
I20260619 14:51:40.957427 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:40.951997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:36127 (local address 127.0.0.1:51004)
0619 14:51:40.952513 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:40.952529 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:40.952634 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:40.953483 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:40.953486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:40.953507 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:40.953619 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:40.953627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.954869 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.954872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:40.955672 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:40.955680 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.956823 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.956828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:40.956947 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:40.956991 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 14:51:40.957117 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 14:51:40.957263 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":5,"thread_start_us":73,"threads_started":1}
I20260619 14:51:40.957501 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:40.951981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58332 (local address 127.19.34.65:40305)
0619 14:51:40.952673 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:40.952685 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:40.952707 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:51:40.952788 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:40.952796 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:40.952997 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:40.953142 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:40.953617 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.954508 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.955822 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.956637 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.956749 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:40.956884 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:40.957257 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 14:51:40.957356 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":104,"threads_started":1}
I20260619 14:51:40.957878 20637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:40.952693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:37517 (local address 127.0.0.1:41864)
0619 14:51:40.952975 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:40.952988 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:40.953071 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:40.954042 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:40.954045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:40.954060 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:40.954133 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:40.954138 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.955210 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.955213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:40.956123 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:40.956144 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.957424 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.957428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:40.957545 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:40.957577 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 14:51:40.957653 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:51:40.957723 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":110,"threads_started":1}
I20260619 14:51:40.958107 20633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:40.951890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:40305 (local address 127.0.0.1:58332)
0619 14:51:40.952166 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:40.952206 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:40.952367 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:40.953336 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:40.953343 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:40.953365 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:40.953490 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:40.953498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.954641 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.954644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:40.955716 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:40.955726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.956946 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.956950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:40.957031 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:40.957048 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 14:51:40.957140 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 14:51:40.957713 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":503,"thread_start_us":66,"threads_started":1}
I20260619 14:51:40.958465 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:40.952953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41864 (local address 127.19.34.67:37517)
0619 14:51:40.953509 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:40.953521 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:40.953542 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:51:40.953605 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:40.953610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:40.953795 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:40.953916 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:40.954250 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.955088 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.956292 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:40.957109 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:40.957228 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:40.957364 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:40.957697 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 14:51:40.958318 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":138,"threads_started":1}
I20260619 14:51:40.960402 20422 tablet_service.cc:1467] Processing CreateTablet for tablet 7b311e41954e402b9150db000daeb927 (DEFAULT_TABLE table=test-table [id=2972b1e362004d78818b6d5cbe4d5152]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:40.960559 20288 tablet_service.cc:1467] Processing CreateTablet for tablet 7b311e41954e402b9150db000daeb927 (DEFAULT_TABLE table=test-table [id=2972b1e362004d78818b6d5cbe4d5152]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:40.960651 20556 tablet_service.cc:1467] Processing CreateTablet for tablet 7b311e41954e402b9150db000daeb927 (DEFAULT_TABLE table=test-table [id=2972b1e362004d78818b6d5cbe4d5152]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:40.961361 20288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b311e41954e402b9150db000daeb927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:40.961361 20422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b311e41954e402b9150db000daeb927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:40.961376 20556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b311e41954e402b9150db000daeb927. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:40.966557 20639 tablet_bootstrap.cc:492] T 7b311e41954e402b9150db000daeb927 P 5835309538e74a7a8116d3fe430e50ed: Bootstrap starting.
I20260619 14:51:40.968463 20640 tablet_bootstrap.cc:492] T 7b311e41954e402b9150db000daeb927 P 6bc1d2eea1d84b6b8e7e3dd5c3e6e68c: Bootstrap starting.
I20260619 14:51:40.968494 20641 tablet_bootstrap.cc:492] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1: Bootstrap starting.
I20260619 14:51:40.969497 20639 tablet_bootstrap.cc:654] T 7b311e41954e402b9150db000daeb927 P 5835309538e74a7a8116d3fe430e50ed: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:40.970328 20639 log.cc:824] T 7b311e41954e402b9150db000daeb927 P 5835309538e74a7a8116d3fe430e50ed: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:40.970825 20641 tablet_bootstrap.cc:654] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:40.970882 20640 tablet_bootstrap.cc:654] T 7b311e41954e402b9150db000daeb927 P 6bc1d2eea1d84b6b8e7e3dd5c3e6e68c: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:40.971578 20641 log.cc:824] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:40.971639 20640 log.cc:824] T 7b311e41954e402b9150db000daeb927 P 6bc1d2eea1d84b6b8e7e3dd5c3e6e68c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:40.972440 20639 tablet_bootstrap.cc:492] T 7b311e41954e402b9150db000daeb927 P 5835309538e74a7a8116d3fe430e50ed: No bootstrap required, opened a new log
I20260619 14:51:40.972604 20639 ts_tablet_manager.cc:1397] T 7b311e41954e402b9150db000daeb927 P 5835309538e74a7a8116d3fe430e50ed: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:51:40.973467 20641 tablet_bootstrap.cc:492] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1: No bootstrap required, opened a new log
I20260619 14:51:40.973476 20640 tablet_bootstrap.cc:492] T 7b311e41954e402b9150db000daeb927 P 6bc1d2eea1d84b6b8e7e3dd5c3e6e68c: No bootstrap required, opened a new log
I20260619 14:51:40.973634 20641 ts_tablet_manager.cc:1397] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:51:40.973649 20640 ts_tablet_manager.cc:1397] T 7b311e41954e402b9150db000daeb927 P 6bc1d2eea1d84b6b8e7e3dd5c3e6e68c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:51:40.976174 20639 raft_consensus.cc:348] T 7b311e41954e402b9150db000daeb927 P 5835309538e74a7a8116d3fe430e50ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc1d2eea1d84b6b8e7e3dd5c3e6e68c" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 40305 } } peers { permanent_uuid: "0ac295d70b6e48eaaaf580e3ab6ccde1" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 36127 } } peers { permanent_uuid: "5835309538e74a7a8116d3fe430e50ed" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 37517 } }
I20260619 14:51:40.976218 20640 raft_consensus.cc:348] T 7b311e41954e402b9150db000daeb927 P 6bc1d2eea1d84b6b8e7e3dd5c3e6e68c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc1d2eea1d84b6b8e7e3dd5c3e6e68c" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 40305 } } peers { permanent_uuid: "0ac295d70b6e48eaaaf580e3ab6ccde1" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 36127 } } peers { permanent_uuid: "5835309538e74a7a8116d3fe430e50ed" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 37517 } }
I20260619 14:51:40.976466 20639 raft_consensus.cc:374] T 7b311e41954e402b9150db000daeb927 P 5835309538e74a7a8116d3fe430e50ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:40.976528 20639 raft_consensus.cc:729] T 7b311e41954e402b9150db000daeb927 P 5835309538e74a7a8116d3fe430e50ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5835309538e74a7a8116d3fe430e50ed, State: Initialized, Role: FOLLOWER
I20260619 14:51:40.976543 20640 raft_consensus.cc:374] T 7b311e41954e402b9150db000daeb927 P 6bc1d2eea1d84b6b8e7e3dd5c3e6e68c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:40.976603 20640 raft_consensus.cc:729] T 7b311e41954e402b9150db000daeb927 P 6bc1d2eea1d84b6b8e7e3dd5c3e6e68c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bc1d2eea1d84b6b8e7e3dd5c3e6e68c, State: Initialized, Role: FOLLOWER
I20260619 14:51:40.976904 20639 consensus_queue.cc:260] T 7b311e41954e402b9150db000daeb927 P 5835309538e74a7a8116d3fe430e50ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc1d2eea1d84b6b8e7e3dd5c3e6e68c" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 40305 } } peers { permanent_uuid: "0ac295d70b6e48eaaaf580e3ab6ccde1" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 36127 } } peers { permanent_uuid: "5835309538e74a7a8116d3fe430e50ed" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 37517 } }
I20260619 14:51:40.976954 20640 consensus_queue.cc:260] T 7b311e41954e402b9150db000daeb927 P 6bc1d2eea1d84b6b8e7e3dd5c3e6e68c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc1d2eea1d84b6b8e7e3dd5c3e6e68c" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 40305 } } peers { permanent_uuid: "0ac295d70b6e48eaaaf580e3ab6ccde1" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 36127 } } peers { permanent_uuid: "5835309538e74a7a8116d3fe430e50ed" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 37517 } }
I20260619 14:51:40.977025 20641 raft_consensus.cc:348] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc1d2eea1d84b6b8e7e3dd5c3e6e68c" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 40305 } } peers { permanent_uuid: "0ac295d70b6e48eaaaf580e3ab6ccde1" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 36127 } } peers { permanent_uuid: "5835309538e74a7a8116d3fe430e50ed" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 37517 } }
I20260619 14:51:40.977268 20641 raft_consensus.cc:374] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:40.977322 20641 raft_consensus.cc:729] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac295d70b6e48eaaaf580e3ab6ccde1, State: Initialized, Role: FOLLOWER
I20260619 14:51:40.977669 20641 consensus_queue.cc:260] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc1d2eea1d84b6b8e7e3dd5c3e6e68c" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 40305 } } peers { permanent_uuid: "0ac295d70b6e48eaaaf580e3ab6ccde1" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 36127 } } peers { permanent_uuid: "5835309538e74a7a8116d3fe430e50ed" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 37517 } }
I20260619 14:51:40.977828 20639 ts_tablet_manager.cc:1428] T 7b311e41954e402b9150db000daeb927 P 5835309538e74a7a8116d3fe430e50ed: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:51:40.977883 20640 ts_tablet_manager.cc:1428] T 7b311e41954e402b9150db000daeb927 P 6bc1d2eea1d84b6b8e7e3dd5c3e6e68c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 14:51:40.978129 20622 heartbeater.cc:503] Master 127.19.34.126:38093 was elected leader, sending a full tablet report...
I20260619 14:51:40.978209 20354 heartbeater.cc:503] Master 127.19.34.126:38093 was elected leader, sending a full tablet report...
I20260619 14:51:40.978475 20641 ts_tablet_manager.cc:1428] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:51:40.978986 20488 heartbeater.cc:503] Master 127.19.34.126:38093 was elected leader, sending a full tablet report...
W20260619 14:51:41.028776 20355 tablet.cc:2369] T 7b311e41954e402b9150db000daeb927 P 6bc1d2eea1d84b6b8e7e3dd5c3e6e68c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:51:41.129339 20647 raft_consensus.cc:482] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:51:41.129528 20647 raft_consensus.cc:504] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc1d2eea1d84b6b8e7e3dd5c3e6e68c" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 40305 } } peers { permanent_uuid: "0ac295d70b6e48eaaaf580e3ab6ccde1" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 36127 } } peers { permanent_uuid: "5835309538e74a7a8116d3fe430e50ed" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 37517 } }
I20260619 14:51:41.130671 20647 leader_election.cc:288] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bc1d2eea1d84b6b8e7e3dd5c3e6e68c (127.19.34.65:40305), 5835309538e74a7a8116d3fe430e50ed (127.19.34.67:37517)
I20260619 14:51:41.134310 20638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:41.131009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:57595 (local address 127.19.34.67:37517)
0619 14:51:41.131114 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:41.131120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:41.131157 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:51:41.131210 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:41.131212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:41.131274 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:41.131357 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:41.131642 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:41.132251 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:41.133347 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:41.133765 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:41.133860 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:41.133928 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:41.134117 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 14:51:41.134185 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:51:41.134315 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:41.130978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:39271 (local address 127.19.34.65:40305)
0619 14:51:41.131089 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:41.131093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:41.131381 (+ 288us) server_negotiation.cc:407] Connection header received
0619 14:51:41.131450 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:41.131453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:41.131500 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:41.131572 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:41.131881 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:41.132419 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:41.133428 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:41.133808 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:41.133927 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:41.133975 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:41.134161 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 14:51:41.134224 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:51:41.134312 20490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:41.130888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:37517 (local address 127.19.34.66:57595)
0619 14:51:41.130983 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:41.130997 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:41.131081 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:41.131397 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:41.131401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:41.131421 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:41.131538 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:41.131545 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:41.132397 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:41.132400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:41.133219 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:41.133228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:41.133892 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:41.133895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:41.134004 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:41.134022 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:51:41.134080 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 14:51:41.134160 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260619 14:51:41.134325 20648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:41.130888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:40305 (local address 127.19.34.66:39271)
0619 14:51:41.131273 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:41.131285 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:41.131379 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:41.131605 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:41.131608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:41.131628 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:41.131772 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:41.131779 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:41.132524 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:41.132526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:41.133308 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:41.133319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:41.133934 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:41.133940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:41.134033 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:41.134047 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 14:51:41.134147 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 14:51:41.134199 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":96,"thread_start_us":94,"threads_started":1}
I20260619 14:51:41.134958 20576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b311e41954e402b9150db000daeb927" candidate_uuid: "0ac295d70b6e48eaaaf580e3ab6ccde1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5835309538e74a7a8116d3fe430e50ed" is_pre_election: true
I20260619 14:51:41.135059 20308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b311e41954e402b9150db000daeb927" candidate_uuid: "0ac295d70b6e48eaaaf580e3ab6ccde1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bc1d2eea1d84b6b8e7e3dd5c3e6e68c" is_pre_election: true
I20260619 14:51:41.135236 20576 raft_consensus.cc:2454] T 7b311e41954e402b9150db000daeb927 P 5835309538e74a7a8116d3fe430e50ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ac295d70b6e48eaaaf580e3ab6ccde1 in term 0.
I20260619 14:51:41.135345 20308 raft_consensus.cc:2454] T 7b311e41954e402b9150db000daeb927 P 6bc1d2eea1d84b6b8e7e3dd5c3e6e68c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ac295d70b6e48eaaaf580e3ab6ccde1 in term 0.
I20260619 14:51:41.135617 20375 leader_election.cc:302] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac295d70b6e48eaaaf580e3ab6ccde1, 5835309538e74a7a8116d3fe430e50ed; no voters:
I20260619 14:51:41.135864 20647 raft_consensus.cc:2781] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:51:41.135933 20647 raft_consensus.cc:482] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:51:41.135970 20647 raft_consensus.cc:3037] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:41.136925 20647 raft_consensus.cc:504] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc1d2eea1d84b6b8e7e3dd5c3e6e68c" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 40305 } } peers { permanent_uuid: "0ac295d70b6e48eaaaf580e3ab6ccde1" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 36127 } } peers { permanent_uuid: "5835309538e74a7a8116d3fe430e50ed" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 37517 } }
I20260619 14:51:41.137204 20647 leader_election.cc:288] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1 [CANDIDATE]: Term 1 election: Requested vote from peers 6bc1d2eea1d84b6b8e7e3dd5c3e6e68c (127.19.34.65:40305), 5835309538e74a7a8116d3fe430e50ed (127.19.34.67:37517)
I20260619 14:51:41.137473 20308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b311e41954e402b9150db000daeb927" candidate_uuid: "0ac295d70b6e48eaaaf580e3ab6ccde1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bc1d2eea1d84b6b8e7e3dd5c3e6e68c"
I20260619 14:51:41.137499 20576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b311e41954e402b9150db000daeb927" candidate_uuid: "0ac295d70b6e48eaaaf580e3ab6ccde1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5835309538e74a7a8116d3fe430e50ed"
I20260619 14:51:41.137591 20308 raft_consensus.cc:3037] T 7b311e41954e402b9150db000daeb927 P 6bc1d2eea1d84b6b8e7e3dd5c3e6e68c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:41.137614 20576 raft_consensus.cc:3037] T 7b311e41954e402b9150db000daeb927 P 5835309538e74a7a8116d3fe430e50ed [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:41.138772 20576 raft_consensus.cc:2454] T 7b311e41954e402b9150db000daeb927 P 5835309538e74a7a8116d3fe430e50ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac295d70b6e48eaaaf580e3ab6ccde1 in term 1.
I20260619 14:51:41.138772 20308 raft_consensus.cc:2454] T 7b311e41954e402b9150db000daeb927 P 6bc1d2eea1d84b6b8e7e3dd5c3e6e68c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac295d70b6e48eaaaf580e3ab6ccde1 in term 1.
I20260619 14:51:41.139024 20375 leader_election.cc:302] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac295d70b6e48eaaaf580e3ab6ccde1, 5835309538e74a7a8116d3fe430e50ed; no voters:
I20260619 14:51:41.139240 20647 raft_consensus.cc:2781] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:41.139391 20647 raft_consensus.cc:686] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1 [term 1 LEADER]: Becoming Leader. State: Replica: 0ac295d70b6e48eaaaf580e3ab6ccde1, State: Running, Role: LEADER
I20260619 14:51:41.139650 20647 consensus_queue.cc:237] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc1d2eea1d84b6b8e7e3dd5c3e6e68c" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 40305 } } peers { permanent_uuid: "0ac295d70b6e48eaaaf580e3ab6ccde1" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 36127 } } peers { permanent_uuid: "5835309538e74a7a8116d3fe430e50ed" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 37517 } }
I20260619 14:51:41.142481 20163 catalog_manager.cc:5495] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ac295d70b6e48eaaaf580e3ab6ccde1 (127.19.34.66). New cstate: current_term: 1 leader_uuid: "0ac295d70b6e48eaaaf580e3ab6ccde1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bc1d2eea1d84b6b8e7e3dd5c3e6e68c" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 40305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ac295d70b6e48eaaaf580e3ab6ccde1" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 36127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5835309538e74a7a8116d3fe430e50ed" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 37517 } health_report { overall_health: UNKNOWN } } }
W20260619 14:51:41.159107 20623 tablet.cc:2369] T 7b311e41954e402b9150db000daeb927 P 5835309538e74a7a8116d3fe430e50ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 14:51:41 dist-test-slave-3qdh krb5kdc[20118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.65@KRBTEST.COM
I20260619 14:51:41.191250 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:41.182163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58346 (local address 127.19.34.65:40305)
0619 14:51:41.182304 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:41.182309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:41.182447 (+ 138us) server_negotiation.cc:407] Connection header received
0619 14:51:41.182581 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:41.182584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:41.182634 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:41.182763 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:41.183495 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:41.184002 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:41.184922 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:41.185150 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:41.185267 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:41.187708 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:41.187723 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:41.187739 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:41.187766 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:41.189392 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:41.189799 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:41.189804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:41.189805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:41.189888 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:41.190193 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:41.190198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:41.190199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:41.190539 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:41.190779 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:41.190943 (+ 164us) server_negotiation.cc:299] Negotiation successful
0619 14:51:41.191066 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 14:51:41 dist-test-slave-3qdh krb5kdc[20118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880701, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:51:41 dist-test-slave-3qdh krb5kdc[20118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880701, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.34.65@KRBTEST.COM
I20260619 14:51:41.214355 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:41.207172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58354 (local address 127.19.34.65:40305)
0619 14:51:41.207278 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:41.207281 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:41.207442 (+ 161us) server_negotiation.cc:407] Connection header received
0619 14:51:41.207512 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:41.207514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:41.207544 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:41.207600 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:41.208136 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:41.208604 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:41.209322 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:41.209509 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:41.209578 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:41.211666 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:41.211669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:41.211670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:41.211688 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:41.213108 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:41.213457 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:41.213460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:41.213461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:41.213525 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:41.213734 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:41.213736 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:41.213737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:41.213868 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:41.213958 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:41.214159 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 14:51:41.214251 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 14:51:41.221094 20489 tablet.cc:2369] T 7b311e41954e402b9150db000daeb927 P 0ac295d70b6e48eaaaf580e3ab6ccde1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 14:51:41 dist-test-slave-3qdh krb5kdc[20118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:51:41 dist-test-slave-3qdh krb5kdc[20118](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.65@KRBTEST.COM
I20260619 14:51:41.235901 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:41.228568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58356 (local address 127.19.34.65:40305)
0619 14:51:41.228691 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:41.228696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:41.228831 (+ 135us) server_negotiation.cc:407] Connection header received
0619 14:51:41.228954 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:41.228957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:41.228995 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:41.229093 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:41.229646 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:41.230245 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:41.231111 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:41.231326 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:41.231425 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:41.233450 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:41.233454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:41.233458 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:41.233483 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:41.234519 (+ 1036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:41.234888 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:41.234893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:41.234894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:41.234963 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:41.235203 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:41.235208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:41.235209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:41.235346 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:41.235475 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:41.235671 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 14:51:41.235766 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 14:51:41.237025 20268 tablet_service.cc:2908] Rejecting scan request for tablet 7b311e41954e402b9150db000daeb927: Uninitialized: clean time has not yet been initialized
I20260619 14:51:41.238801 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 20225
I20260619 14:51:41.244124 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 20359
I20260619 14:51:41.249229 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 20493
I20260619 14:51:41.253728 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 20131
2026-06-19T14:51:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2268 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:41 dist-test-slave-3qdh krb5kdc[20663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:41 dist-test-slave-3qdh krb5kdc[20663](info): set up 2 sockets
Jun 19 14:51:41 dist-test-slave-3qdh krb5kdc[20663](info): commencing operation
krb5kdc: starting...
W20260619 14:51:42.231787 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:51:42 dist-test-slave-3qdh krb5kdc[20663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:51:42.336992 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:42433
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:42433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:42.433712 20676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:42.433961 20676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:42.434031 20676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:42.437832 20676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:42.437901 20676 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 14:51:42.437928 20676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:42.437968 20676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:42.437997 20676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:42.438027 20676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:42.443225 20676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:42433
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:42433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:42.444095 20676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:42.444993 20676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:42.452615 20681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:42.452625 20684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:42.452606 20682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:42.452966 20676 server_base.cc:1034] running on GCE node
I20260619 14:51:42.453459 20676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:42.454406 20676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:42.455583 20676 hybrid_clock.cc:648] HybridClock initialized: now 1781880702455559 us; error 42 us; skew 500 ppm
Jun 19 14:51:42 dist-test-slave-3qdh krb5kdc[20663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880702, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:42.462455 20676 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:51:42.463306 20676 webserver.cc:460] Webserver started at http://127.19.34.126:33565/ using document root <none> and password file <none>
I20260619 14:51:42.463768 20676 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:42.463821 20676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:42.463985 20676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:42.465787 20676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "e8f103ab00ac4cec847d60ba0e167ca8"
format_stamp: "Formatted at 2026-06-19 14:51:42 on dist-test-slave-3qdh"
server_key: "aaa641e5d29d0e99a981a0ad3d9dac2f"
server_key_iv: "1092cef546dc3911eedb0acfffa6b051"
server_key_version: "encryptionkey@0"
I20260619 14:51:42.466284 20676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "e8f103ab00ac4cec847d60ba0e167ca8"
format_stamp: "Formatted at 2026-06-19 14:51:42 on dist-test-slave-3qdh"
server_key: "aaa641e5d29d0e99a981a0ad3d9dac2f"
server_key_iv: "1092cef546dc3911eedb0acfffa6b051"
server_key_version: "encryptionkey@0"
I20260619 14:51:42.469640 20676 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 14:51:42.471681 20691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:42.472823 20676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:42.472941 20676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "e8f103ab00ac4cec847d60ba0e167ca8"
format_stamp: "Formatted at 2026-06-19 14:51:42 on dist-test-slave-3qdh"
server_key: "aaa641e5d29d0e99a981a0ad3d9dac2f"
server_key_iv: "1092cef546dc3911eedb0acfffa6b051"
server_key_version: "encryptionkey@0"
I20260619 14:51:42.473037 20676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:42.501621 20676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:42.504192 20676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:42.504395 20676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:42.511418 20676 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:42433
I20260619 14:51:42.511444 20753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:42433 every 8 connection(s)
I20260619 14:51:42.512256 20676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:42.514681 20754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:42.519706 20754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8: Bootstrap starting.
I20260619 14:51:42.521813 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 20676
I20260619 14:51:42.521919 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:51:42.522104 20754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:42.522194 19593 external_mini_cluster.cc:1428] Setting key 808c6bcff8b724b383ab8a8717b78605
I20260619 14:51:42.522883 20754 log.cc:824] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:42.524708 20754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8: No bootstrap required, opened a new log
I20260619 14:51:42.528458 20754 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f103ab00ac4cec847d60ba0e167ca8" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42433 } }
I20260619 14:51:42.528739 20754 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:42.528805 20754 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f103ab00ac4cec847d60ba0e167ca8, State: Initialized, Role: FOLLOWER
I20260619 14:51:42.529289 20754 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f103ab00ac4cec847d60ba0e167ca8" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42433 } }
I20260619 14:51:42.529443 20754 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:42.529516 20754 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:42.529630 20754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 14:51:42 dist-test-slave-3qdh krb5kdc[20663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:42.530434 20754 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f103ab00ac4cec847d60ba0e167ca8" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42433 } }
I20260619 14:51:42.530709 20754 leader_election.cc:302] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8f103ab00ac4cec847d60ba0e167ca8; no voters:
I20260619 14:51:42.530951 20754 leader_election.cc:288] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:42.531175 20759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:42.531390 20759 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8 [term 1 LEADER]: Becoming Leader. State: Replica: e8f103ab00ac4cec847d60ba0e167ca8, State: Running, Role: LEADER
I20260619 14:51:42.531701 20759 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f103ab00ac4cec847d60ba0e167ca8" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42433 } }
I20260619 14:51:42.532063 20754 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:42.533273 20760 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8f103ab00ac4cec847d60ba0e167ca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f103ab00ac4cec847d60ba0e167ca8" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42433 } } }
I20260619 14:51:42.533679 20760 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:42.533564 20761 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8f103ab00ac4cec847d60ba0e167ca8. Latest consensus state: current_term: 1 leader_uuid: "e8f103ab00ac4cec847d60ba0e167ca8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f103ab00ac4cec847d60ba0e167ca8" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42433 } } }
I20260619 14:51:42.533989 20761 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:42.534785 20772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:42.535565 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:42.523237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51240 (local address 127.19.34.126:42433)
0619 14:51:42.523595 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:42.523610 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:42.523641 (+ 31us) server_negotiation.cc:407] Connection header received
0619 14:51:42.524198 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:42.524212 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:42.524563 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:42.524865 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:42.525594 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:42.526446 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:42.527552 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:42.527971 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:42.528070 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:42.530591 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:42.530596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:42.530614 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:42.530634 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:42.532570 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:42.533141 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:42.533147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:42.533152 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:42.533293 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:42.533592 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:42.533598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:42.533603 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:42.534118 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:42.534289 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:42.534534 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 14:51:42.534807 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":109,"threads_started":1}
I20260619 14:51:42.537153 20772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:42.541636 20772 catalog_manager.cc:1347] Generated new cluster ID: ae173392eacd4c41b5b5409e658c539a
I20260619 14:51:42.541741 20772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:42.548964 20772 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:42.549670 20772 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:42.558122 20772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8: Generated new TSK 0
I20260619 14:51:42.558620 20772 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.34.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
I20260619 14:51:42.583015 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:42433
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:42.682826 20780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:42.683044 20780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:42.683112 20780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:42.686894 20780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:42.686939 20780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:42.687011 20780 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:51:42.687083 20780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:51:42.691694 20780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.34.126:42433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:42.692523 20780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:42.693427 20780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:42.701987 20788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:42.702033 20785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:42.702018 20786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:42.702203 20780 server_base.cc:1034] running on GCE node
I20260619 14:51:42.702607 20780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:42.703169 20780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:42.704357 20780 hybrid_clock.cc:648] HybridClock initialized: now 1781880702704339 us; error 30 us; skew 500 ppm
Jun 19 14:51:42 dist-test-slave-3qdh krb5kdc[20663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880702, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:42.706676 20780 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:51:42.707412 20780 webserver.cc:460] Webserver started at http://127.19.34.65:39253/ using document root <none> and password file <none>
I20260619 14:51:42.707890 20780 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:42.707990 20780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:42.708195 20780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:42.709982 20780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "fd3a09b3febe464db009fe162a1d97b8"
format_stamp: "Formatted at 2026-06-19 14:51:42 on dist-test-slave-3qdh"
server_key: "4eda258550df07120f1c34eb1a0fbac8"
server_key_iv: "77b19a080da7ec11d372022083e2d7c3"
server_key_version: "encryptionkey@0"
I20260619 14:51:42.710472 20780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "fd3a09b3febe464db009fe162a1d97b8"
format_stamp: "Formatted at 2026-06-19 14:51:42 on dist-test-slave-3qdh"
server_key: "4eda258550df07120f1c34eb1a0fbac8"
server_key_iv: "77b19a080da7ec11d372022083e2d7c3"
server_key_version: "encryptionkey@0"
I20260619 14:51:42.713675 20780 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:51:42.715723 20795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:42.716835 20780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:42.716941 20780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "fd3a09b3febe464db009fe162a1d97b8"
format_stamp: "Formatted at 2026-06-19 14:51:42 on dist-test-slave-3qdh"
server_key: "4eda258550df07120f1c34eb1a0fbac8"
server_key_iv: "77b19a080da7ec11d372022083e2d7c3"
server_key_version: "encryptionkey@0"
I20260619 14:51:42.717032 20780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:42.734064 20780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:42.736595 20780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:42.736778 20780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:42.738088 20780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:42.738131 20780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:42.738176 20780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:42.738224 20780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:51:42 dist-test-slave-3qdh krb5kdc[20663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880702, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:42.747848 20780 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:35021
I20260619 14:51:42.747953 20910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:35021 every 8 connection(s)
I20260619 14:51:42.748704 20780 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:51:42.749991 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:42.738798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:60471 (local address 127.19.34.126:42433)
0619 14:51:42.738951 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:42.738957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:42.739630 (+ 673us) server_negotiation.cc:407] Connection header received
0619 14:51:42.740493 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:42.740499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:42.740560 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:42.740655 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:42.741759 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:42.742524 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:42.743781 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:42.744101 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:42.744220 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:42.746613 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:42.746619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:42.746623 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:42.746650 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:42.748188 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:42.748697 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:42.748701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:42.748703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:42.748782 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:42.749083 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:42.749086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:42.749088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:42.749291 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:42.749378 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:42.749805 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 14:51:42.749891 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:51:42.750631 20804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:42.739091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:42433 (local address 127.19.34.65:60471)
0619 14:51:42.739485 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:42.739526 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:42.740233 (+ 707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:42.740828 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:42.740836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:42.741144 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:42.741611 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:42.741624 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:42.742662 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:42.742680 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:42.743647 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:42.743656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:42.744261 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:42.744265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:42.744451 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:42.744950 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:42.744973 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:42.746485 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:42.748348 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:42.748353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:42.748365 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:42.748595 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:42.748866 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:42.748869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:42.748871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:42.748991 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:42.749422 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:42.749428 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:42.749598 (+ 170us) client_negotiation.cc:769] Sending connection context
0619 14:51:42.749727 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 14:51:42.749912 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":110,"threads_started":1}
I20260619 14:51:42.751770 20911 heartbeater.cc:347] Connected to a master server at 127.19.34.126:42433
I20260619 14:51:42.751893 20911 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:42.752288 20911 heartbeater.cc:511] Master 127.19.34.126:42433 requested a full tablet report, sending...
I20260619 14:51:42.753921 20707 ts_manager.cc:194] Registered new tserver with Master: fd3a09b3febe464db009fe162a1d97b8 (127.19.34.65:35021)
I20260619 14:51:42.755254 20707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:60471
I20260619 14:51:42.758354 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 20780
I20260619 14:51:42.758455 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:51:42.758733 19593 external_mini_cluster.cc:1428] Setting key 64f00faf7af52d3825361ec1302590e2
I20260619 14:51:42.760316 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:42.753798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:36353 (local address 127.19.34.126:42433)
0619 14:51:42.753911 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:42.753915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:42.754152 (+ 237us) server_negotiation.cc:407] Connection header received
0619 14:51:42.754268 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:42.754270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:42.754302 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:42.754380 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:42.754826 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:42.755294 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:42.756292 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:42.756598 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:42.756691 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:42.757547 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:42.757549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:42.757551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:42.757567 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:42.758663 (+ 1096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:42.759069 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:42.759072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:42.759073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:42.759145 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:42.759531 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:42.759533 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:42.759534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:42.759672 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:42.759743 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:42.760029 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 14:51:42.760127 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:51:42.760367 20917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:42.753684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:42433 (local address 127.19.34.65:36353)
0619 14:51:42.754079 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:42.754092 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:42.754184 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:42.754424 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:42.754426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:42.754609 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:42.754724 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:42.754731 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:42.755438 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:42.755442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:42.756194 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:42.756201 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:42.756698 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:42.756700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:42.756755 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:42.757171 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:42.757181 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:42.757444 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:42.758755 (+ 1311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:42.758757 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:42.758758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:42.758975 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:42.759226 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:42.759229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:42.759231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:42.759437 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:42.759757 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:42.759760 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:42.759841 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 14:51:42.759971 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 14:51:42.760120 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.19.34.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
I20260619 14:51:42.780978 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:42433
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:42.883505 20920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:42.883750 20920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:42.883822 20920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:42.887647 20920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:42.887729 20920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:42.887809 20920 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:51:42.887874 20920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:51:42.892264 20920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.34.126:42433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:42.893124 20920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:42.893981 20920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:42.901993 20928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:42.902025 20925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:42.902093 20920 server_base.cc:1034] running on GCE node
W20260619 14:51:42.902017 20926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:42.902532 20920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:42.903102 20920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:42.904248 20920 hybrid_clock.cc:648] HybridClock initialized: now 1781880702904233 us; error 28 us; skew 500 ppm
Jun 19 14:51:42 dist-test-slave-3qdh krb5kdc[20663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880702, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:42.906518 20920 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:51:42.907194 20920 webserver.cc:460] Webserver started at http://127.19.34.66:44465/ using document root <none> and password file <none>
I20260619 14:51:42.907644 20920 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:42.907696 20920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:42.907928 20920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:42.909704 20920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "d4d6ad23acd4443a93d794f72dc3626f"
format_stamp: "Formatted at 2026-06-19 14:51:42 on dist-test-slave-3qdh"
server_key: "8da6b79b41a0bf868dd5668999b48cc5"
server_key_iv: "4a9a86734c02d10699a79b6cbd192cde"
server_key_version: "encryptionkey@0"
I20260619 14:51:42.910216 20920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "d4d6ad23acd4443a93d794f72dc3626f"
format_stamp: "Formatted at 2026-06-19 14:51:42 on dist-test-slave-3qdh"
server_key: "8da6b79b41a0bf868dd5668999b48cc5"
server_key_iv: "4a9a86734c02d10699a79b6cbd192cde"
server_key_version: "encryptionkey@0"
I20260619 14:51:42.913347 20920 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:51:42.915489 20935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:42.916636 20920 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:51:42.916755 20920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "d4d6ad23acd4443a93d794f72dc3626f"
format_stamp: "Formatted at 2026-06-19 14:51:42 on dist-test-slave-3qdh"
server_key: "8da6b79b41a0bf868dd5668999b48cc5"
server_key_iv: "4a9a86734c02d10699a79b6cbd192cde"
server_key_version: "encryptionkey@0"
I20260619 14:51:42.916848 20920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:42.927104 20920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:42.929431 20920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:42.929618 20920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:42.930676 20920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:42.930755 20920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:42.930816 20920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:42.930847 20920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:51:42 dist-test-slave-3qdh krb5kdc[20663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880702, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:42.941006 20920 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:34803
I20260619 14:51:42.941015 21050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:34803 every 8 connection(s)
I20260619 14:51:42.941964 20920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:51:42.943032 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:42.931372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:41365 (local address 127.19.34.126:42433)
0619 14:51:42.931482 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:42.931485 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:42.932293 (+ 808us) server_negotiation.cc:407] Connection header received
0619 14:51:42.933334 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:42.933337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:42.933395 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:42.933499 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:42.934668 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:42.935497 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:42.936694 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:42.936972 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:42.937071 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:42.939382 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:42.939386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:42.939388 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:42.939410 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:42.941122 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:42.941614 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:42.941616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:42.941618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:42.941684 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:42.942045 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:42.942051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:42.942053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:42.942236 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:42.942329 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:42.942830 (+ 501us) server_negotiation.cc:299] Negotiation successful
0619 14:51:42.942922 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:51:42.943658 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:42.931762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:42433 (local address 127.19.34.66:41365)
0619 14:51:42.932177 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:42.932211 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:42.933127 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:42.933785 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:42.933795 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:42.934151 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:42.934536 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:42.934546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:42.935613 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:42.935619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:42.936588 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:42.936599 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:42.937102 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:42.937106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:42.937262 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:42.937771 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:42.937792 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:42.939248 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:42.941242 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:42.941247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:42.941257 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:42.941524 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:42.941777 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:42.941779 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:42.941781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:42.941934 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:42.942374 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:42.942379 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:42.942601 (+ 222us) client_negotiation.cc:769] Sending connection context
0619 14:51:42.942761 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 14:51:42.942976 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":97,"threads_started":1}
I20260619 14:51:42.944805 21051 heartbeater.cc:347] Connected to a master server at 127.19.34.126:42433
I20260619 14:51:42.944940 21051 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:42.945330 21051 heartbeater.cc:511] Master 127.19.34.126:42433 requested a full tablet report, sending...
I20260619 14:51:42.945622 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 20920
I20260619 14:51:42.945784 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:51:42.946039 19593 external_mini_cluster.cc:1428] Setting key a78c9db16b8a95aca7ff4ca3b39ea6ef
I20260619 14:51:42.946273 20708 ts_manager.cc:194] Registered new tserver with Master: d4d6ad23acd4443a93d794f72dc3626f (127.19.34.66:34803)
I20260619 14:51:42.946974 20708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:41365
I20260619 14:51:42.953088 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:42.947016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:33587 (local address 127.19.34.126:42433)
0619 14:51:42.947171 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:42.947174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:42.947186 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:51:42.947224 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:42.947227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:42.947260 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:42.947335 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:42.947666 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:42.948109 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:42.948901 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:42.949142 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:42.949256 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:42.950120 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:42.950123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:42.950127 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:42.950151 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:42.951460 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:42.951858 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:42.951862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:42.951864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:42.951967 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:42.952259 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:42.952263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:42.952264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:42.952484 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:42.952587 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:42.952840 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 14:51:42.952939 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 14:51:42.953120 21057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:42.946722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:42433 (local address 127.19.34.66:33587)
0619 14:51:42.946986 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:42.946999 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:42.947072 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:42.947358 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:42.947360 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:42.947490 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:42.947565 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:42.947569 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:42.948223 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:42.948226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:42.948794 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:42.948801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:42.949266 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:42.949269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:42.949347 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:42.949741 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:42.949751 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:42.950000 (+ 249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:42.951558 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:42.951563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:42.951565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:42.951744 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:42.952061 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:42.952065 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:42.952067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:42.952167 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:42.952580 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:42.952583 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:42.952673 (+ 90us) client_negotiation.cc:769] Sending connection context
0619 14:51:42.952790 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 14:51:42.952911 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.19.34.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
I20260619 14:51:42.970582 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:42433
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:41389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:43.069304 21060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:43.069505 21060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:43.069572 21060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:43.073350 21060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:43.073395 21060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:43.073459 21060 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:51:43.073540 21060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:51:43.078279 21060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:41389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.34.126:42433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:43.079036 21060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:43.079821 21060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:43.087809 21066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:43.087782 21065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:43.087788 21068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:43.088048 21060 server_base.cc:1034] running on GCE node
I20260619 14:51:43.088636 21060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:43.089192 21060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:43.090369 21060 hybrid_clock.cc:648] HybridClock initialized: now 1781880703090351 us; error 28 us; skew 500 ppm
Jun 19 14:51:43 dist-test-slave-3qdh krb5kdc[20663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880703, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:43.092916 21060 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:51:43.093678 21060 webserver.cc:460] Webserver started at http://127.19.34.67:45719/ using document root <none> and password file <none>
I20260619 14:51:43.094138 21060 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:43.094187 21060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:43.094317 21060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:43.096006 21060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "4b630441204d4e5f8724d50b3a927a53"
format_stamp: "Formatted at 2026-06-19 14:51:43 on dist-test-slave-3qdh"
server_key: "93dfe606d0747eed3a4b3c1324fbaf48"
server_key_iv: "84ad4729317bae0710aa3603dc474fcd"
server_key_version: "encryptionkey@0"
I20260619 14:51:43.096503 21060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "4b630441204d4e5f8724d50b3a927a53"
format_stamp: "Formatted at 2026-06-19 14:51:43 on dist-test-slave-3qdh"
server_key: "93dfe606d0747eed3a4b3c1324fbaf48"
server_key_iv: "84ad4729317bae0710aa3603dc474fcd"
server_key_version: "encryptionkey@0"
I20260619 14:51:43.099547 21060 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:51:43.101603 21075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:43.102623 21060 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260619 14:51:43.102708 21060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "4b630441204d4e5f8724d50b3a927a53"
format_stamp: "Formatted at 2026-06-19 14:51:43 on dist-test-slave-3qdh"
server_key: "93dfe606d0747eed3a4b3c1324fbaf48"
server_key_iv: "84ad4729317bae0710aa3603dc474fcd"
server_key_version: "encryptionkey@0"
I20260619 14:51:43.102778 21060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:43.118873 21060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:43.121924 21060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:43.122085 21060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:43.123303 21060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:43.123347 21060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:43.123383 21060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:43.123433 21060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:51:43 dist-test-slave-3qdh krb5kdc[20663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880703, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:43.133399 21060 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:44369
I20260619 14:51:43.134083 21190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:44369 every 8 connection(s)
I20260619 14:51:43.134228 21060 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:51:43.135170 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 21060
I20260619 14:51:43.135280 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:51:43.135475 19593 external_mini_cluster.cc:1428] Setting key b9f5cc2cfa5e54c7106116390ed18562
I20260619 14:51:43.135924 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.123911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:38315 (local address 127.19.34.126:42433)
0619 14:51:43.124095 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:43.124100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:43.125328 (+ 1228us) server_negotiation.cc:407] Connection header received
0619 14:51:43.126352 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:43.126355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:43.126408 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:43.126483 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:43.127972 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.128527 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.129735 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.129991 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.130093 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.132480 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:43.132483 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:43.132486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:43.132514 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:43.133944 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:43.134448 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:43.134451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:43.134452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:43.134533 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:43.134883 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:43.134886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:43.134887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:43.135103 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:43.135210 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:43.135647 (+ 437us) server_negotiation.cc:299] Negotiation successful
0619 14:51:43.135764 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:51:43.136722 21085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.124693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:42433 (local address 127.19.34.67:38315)
0619 14:51:43.125193 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:43.125245 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:43.126167 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:43.126682 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:43.126690 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:43.127144 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:43.127813 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.127848 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.128638 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.128642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.129605 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.129614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.130130 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.130134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.130284 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.130738 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:43.130753 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:43.132339 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:43.134064 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:43.134068 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:43.134079 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:43.134365 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:43.134634 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:43.134637 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:43.134638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:43.134782 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:43.135208 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:43.135214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:43.135428 (+ 214us) client_negotiation.cc:769] Sending connection context
0619 14:51:43.135594 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 14:51:43.135863 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":134,"threads_started":1}
I20260619 14:51:43.137737 21191 heartbeater.cc:347] Connected to a master server at 127.19.34.126:42433
I20260619 14:51:43.137861 21191 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:43.138242 21191 heartbeater.cc:511] Master 127.19.34.126:42433 requested a full tablet report, sending...
I20260619 14:51:43.139118 20707 ts_manager.cc:194] Registered new tserver with Master: 4b630441204d4e5f8724d50b3a927a53 (127.19.34.67:44369)
I20260619 14:51:43.139894 20707 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:38315
I20260619 14:51:43.145187 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.139594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:53677 (local address 127.19.34.126:42433)
0619 14:51:43.139701 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:43.139704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:43.139836 (+ 132us) server_negotiation.cc:407] Connection header received
0619 14:51:43.139933 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:43.139934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:43.139987 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:43.140065 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:43.140565 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.140972 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.141650 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.141836 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.141927 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.142677 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:43.142680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:43.142682 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:43.142707 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:43.143788 (+ 1081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:43.144203 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:43.144206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:43.144207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:43.144272 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:43.144544 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:43.144548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:43.144549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:43.144681 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:43.144779 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:43.145006 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 14:51:43.145086 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:51:43.145195 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.139487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:42433 (local address 127.19.34.67:53677)
0619 14:51:43.139755 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:43.139768 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:43.139856 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:43.140080 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:43.140083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:43.140313 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:43.140426 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.140433 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.141080 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.141082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.141550 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.141557 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.141929 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.141932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.142008 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.142338 (+ 330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:43.142347 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:43.142566 (+ 219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:43.143895 (+ 1329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:43.143900 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:43.143901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:43.144104 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:43.144390 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:43.144392 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:43.144394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:43.144466 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:43.144772 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:43.144775 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:43.144847 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 14:51:43.144946 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 14:51:43.145054 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":93,"threads_started":1}
I20260619 14:51:43.148947 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:51:43 dist-test-slave-3qdh krb5kdc[20663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880703, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:51:43 dist-test-slave-3qdh krb5kdc[20663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880703, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:43.170610 20758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.163915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51246 (local address 127.19.34.126:42433)
0619 14:51:43.164023 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:43.164026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:43.164036 (+ 10us) server_negotiation.cc:407] Connection header received
0619 14:51:43.164075 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:43.164077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:43.164105 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:43.164177 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:43.164763 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.165152 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.165794 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.165956 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.166016 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.167900 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:43.167903 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:43.167904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:43.167942 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:43.169305 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:43.169655 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:43.169658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:43.169659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:43.169732 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:43.169938 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:43.169940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:43.169941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:43.170075 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:43.170163 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:43.170361 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 14:51:43.170430 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:51:43.173005 20707 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:51:43.175853 20707 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:51:43.187965 21210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.183360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:35021 (local address 127.0.0.1:57956)
0619 14:51:43.183696 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:43.183710 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:43.183787 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:43.184579 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:43.184581 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:43.184595 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:43.184699 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.184705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.185714 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.185717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.186477 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.186484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.187467 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.187471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.187586 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.187624 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 14:51:43.187699 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 14:51:43.187820 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":84,"threads_started":1}
I20260619 14:51:43.188102 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.183105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38580 (local address 127.19.34.67:44369)
0619 14:51:43.183475 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:43.183487 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:43.183512 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:51:43.183583 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:43.183588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:43.183800 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:43.183949 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:43.184537 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.185538 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.186770 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.187489 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.187586 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.187746 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:43.187848 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 14:51:43.187942 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":97,"threads_started":1}
I20260619 14:51:43.188130 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.182947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:44369 (local address 127.0.0.1:38580)
0619 14:51:43.183254 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:43.183301 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:43.183407 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:43.184125 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:43.184132 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:43.184156 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:43.184419 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.184427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.185647 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.185651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.186662 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.186670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.187602 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.187606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.187702 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.187727 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:51:43.187823 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 14:51:43.187907 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":88,"threads_started":1}
I20260619 14:51:43.188776 21211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.183611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:34803 (local address 127.0.0.1:46336)
0619 14:51:43.183956 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:43.183968 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:43.184052 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:43.184828 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:43.184830 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:43.184846 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:43.184909 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.184914 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.186016 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.186019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.186988 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.187001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.188509 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.188512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.188573 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.188590 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 14:51:43.188637 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 14:51:43.188681 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":79,"threads_started":1}
I20260619 14:51:43.188781 21212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.183771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57956 (local address 127.19.34.65:35021)
0619 14:51:43.184168 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:43.184179 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:43.184197 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:51:43.184266 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:43.184269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:43.184449 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:43.184840 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:43.184944 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.185556 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.186583 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.187328 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.188397 (+ 1069us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.188536 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:43.188580 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 14:51:43.188654 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":98,"threads_started":1}
I20260619 14:51:43.189917 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.184025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.19.34.66:34803)
0619 14:51:43.184442 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:43.184454 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:43.184474 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:51:43.184536 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:43.184540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:43.184691 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:43.184812 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:43.185009 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.185917 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.187120 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.187968 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.189541 (+ 1573us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.189675 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:43.189728 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 14:51:43.189801 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":107,"threads_started":1}
I20260619 14:51:43.190516 21125 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed0e3a8c80a49e884beca712865f8e2 (DEFAULT_TABLE table=test-table [id=3b80c698f6854353bca707561c2f6252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:43.191107 20845 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed0e3a8c80a49e884beca712865f8e2 (DEFAULT_TABLE table=test-table [id=3b80c698f6854353bca707561c2f6252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:43.191478 21125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed0e3a8c80a49e884beca712865f8e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:43.191847 20845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed0e3a8c80a49e884beca712865f8e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:43.192085 20977 tablet_service.cc:1467] Processing CreateTablet for tablet 1ed0e3a8c80a49e884beca712865f8e2 (DEFAULT_TABLE table=test-table [id=3b80c698f6854353bca707561c2f6252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:43.192977 20977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ed0e3a8c80a49e884beca712865f8e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:43.197695 21214 tablet_bootstrap.cc:492] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8: Bootstrap starting.
I20260619 14:51:43.198681 21215 tablet_bootstrap.cc:492] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53: Bootstrap starting.
I20260619 14:51:43.199361 21216 tablet_bootstrap.cc:492] T 1ed0e3a8c80a49e884beca712865f8e2 P d4d6ad23acd4443a93d794f72dc3626f: Bootstrap starting.
I20260619 14:51:43.200001 21214 tablet_bootstrap.cc:654] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:43.200546 21215 tablet_bootstrap.cc:654] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:43.201252 21216 tablet_bootstrap.cc:654] T 1ed0e3a8c80a49e884beca712865f8e2 P d4d6ad23acd4443a93d794f72dc3626f: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:43.201292 21215 log.cc:824] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:43.201399 21214 log.cc:824] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:43.201989 21216 log.cc:824] T 1ed0e3a8c80a49e884beca712865f8e2 P d4d6ad23acd4443a93d794f72dc3626f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:43.203068 21215 tablet_bootstrap.cc:492] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53: No bootstrap required, opened a new log
I20260619 14:51:43.203107 21214 tablet_bootstrap.cc:492] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8: No bootstrap required, opened a new log
I20260619 14:51:43.203260 21215 ts_tablet_manager.cc:1397] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:51:43.203289 21214 ts_tablet_manager.cc:1397] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260619 14:51:43.203714 21216 tablet_bootstrap.cc:492] T 1ed0e3a8c80a49e884beca712865f8e2 P d4d6ad23acd4443a93d794f72dc3626f: No bootstrap required, opened a new log
I20260619 14:51:43.203869 21216 ts_tablet_manager.cc:1397] T 1ed0e3a8c80a49e884beca712865f8e2 P d4d6ad23acd4443a93d794f72dc3626f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:51:43.206346 21215 raft_consensus.cc:348] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } }
I20260619 14:51:43.206647 21215 raft_consensus.cc:374] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:43.206702 21215 raft_consensus.cc:729] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b630441204d4e5f8724d50b3a927a53, State: Initialized, Role: FOLLOWER
I20260619 14:51:43.206928 21214 raft_consensus.cc:348] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } }
I20260619 14:51:43.207154 21214 raft_consensus.cc:374] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:43.207139 21215 consensus_queue.cc:260] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } }
I20260619 14:51:43.207222 21214 raft_consensus.cc:729] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3a09b3febe464db009fe162a1d97b8, State: Initialized, Role: FOLLOWER
I20260619 14:51:43.207396 21216 raft_consensus.cc:348] T 1ed0e3a8c80a49e884beca712865f8e2 P d4d6ad23acd4443a93d794f72dc3626f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } }
I20260619 14:51:43.207592 21216 raft_consensus.cc:374] T 1ed0e3a8c80a49e884beca712865f8e2 P d4d6ad23acd4443a93d794f72dc3626f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:43.207645 21216 raft_consensus.cc:729] T 1ed0e3a8c80a49e884beca712865f8e2 P d4d6ad23acd4443a93d794f72dc3626f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d6ad23acd4443a93d794f72dc3626f, State: Initialized, Role: FOLLOWER
I20260619 14:51:43.207598 21214 consensus_queue.cc:260] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } }
I20260619 14:51:43.208062 21215 ts_tablet_manager.cc:1428] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:51:43.208022 21216 consensus_queue.cc:260] T 1ed0e3a8c80a49e884beca712865f8e2 P d4d6ad23acd4443a93d794f72dc3626f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } }
I20260619 14:51:43.208402 21191 heartbeater.cc:503] Master 127.19.34.126:42433 was elected leader, sending a full tablet report...
I20260619 14:51:43.208453 21214 ts_tablet_manager.cc:1428] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:51:43.208626 21051 heartbeater.cc:503] Master 127.19.34.126:42433 was elected leader, sending a full tablet report...
I20260619 14:51:43.208626 20911 heartbeater.cc:503] Master 127.19.34.126:42433 was elected leader, sending a full tablet report...
I20260619 14:51:43.209954 21216 ts_tablet_manager.cc:1428] T 1ed0e3a8c80a49e884beca712865f8e2 P d4d6ad23acd4443a93d794f72dc3626f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260619 14:51:43.249810 20912 tablet.cc:2369] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:51:43.324517 21220 raft_consensus.cc:482] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:51:43.324709 21220 raft_consensus.cc:504] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } }
I20260619 14:51:43.325728 21220 leader_election.cc:288] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4d6ad23acd4443a93d794f72dc3626f (127.19.34.66:34803), fd3a09b3febe464db009fe162a1d97b8 (127.19.34.65:35021)
I20260619 14:51:43.329830 21212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.326024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:47889 (local address 127.19.34.65:35021)
0619 14:51:43.326133 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:43.326138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:43.326232 (+ 94us) server_negotiation.cc:407] Connection header received
0619 14:51:43.326356 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:43.326359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:43.326420 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:43.326491 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:43.326826 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.327599 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.328763 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.329356 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.329441 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.329480 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:43.329688 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 14:51:43.329739 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:51:43.329854 21085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.325970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:35021 (local address 127.19.34.67:47889)
0619 14:51:43.326144 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:43.326158 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:43.326262 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:43.326519 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:43.326523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:43.326540 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:43.326710 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.326719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.327701 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.327705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.328647 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.328655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.329454 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.329457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.329564 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.329589 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:51:43.329662 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 14:51:43.329726 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 14:51:43.330425 21223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.325970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:34803 (local address 127.19.34.67:55361)
0619 14:51:43.326361 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:43.326374 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:43.326478 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:43.326791 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:43.326794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:43.326812 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:43.327379 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.327388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.328267 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.328269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.329212 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.329222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.330084 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.330087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.330211 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.330236 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:51:43.330291 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 14:51:43.330332 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":29,"thread_start_us":98,"threads_started":1}
I20260619 14:51:43.330479 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.326028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:55361 (local address 127.19.34.66:34803)
0619 14:51:43.326134 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:43.326138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:43.326478 (+ 340us) server_negotiation.cc:407] Connection header received
0619 14:51:43.326577 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:43.326580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:43.326642 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:43.326747 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:43.327495 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.328173 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.329332 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.329947 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.330070 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.330115 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:43.330325 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 14:51:43.330388 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:51:43.330483 20865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed0e3a8c80a49e884beca712865f8e2" candidate_uuid: "4b630441204d4e5f8724d50b3a927a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd3a09b3febe464db009fe162a1d97b8" is_pre_election: true
I20260619 14:51:43.330736 20865 raft_consensus.cc:2454] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b630441204d4e5f8724d50b3a927a53 in term 0.
I20260619 14:51:43.331012 20994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed0e3a8c80a49e884beca712865f8e2" candidate_uuid: "4b630441204d4e5f8724d50b3a927a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d6ad23acd4443a93d794f72dc3626f" is_pre_election: true
I20260619 14:51:43.331187 21077 leader_election.cc:302] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b630441204d4e5f8724d50b3a927a53, fd3a09b3febe464db009fe162a1d97b8; no voters:
I20260619 14:51:43.331288 20994 raft_consensus.cc:2454] T 1ed0e3a8c80a49e884beca712865f8e2 P d4d6ad23acd4443a93d794f72dc3626f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b630441204d4e5f8724d50b3a927a53 in term 0.
I20260619 14:51:43.331434 21220 raft_consensus.cc:2781] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:51:43.331503 21220 raft_consensus.cc:482] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:51:43.331540 21220 raft_consensus.cc:3037] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:43.332696 21220 raft_consensus.cc:504] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } }
I20260619 14:51:43.333086 21220 leader_election.cc:288] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 [CANDIDATE]: Term 1 election: Requested vote from peers d4d6ad23acd4443a93d794f72dc3626f (127.19.34.66:34803), fd3a09b3febe464db009fe162a1d97b8 (127.19.34.65:35021)
I20260619 14:51:43.333356 20994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed0e3a8c80a49e884beca712865f8e2" candidate_uuid: "4b630441204d4e5f8724d50b3a927a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d6ad23acd4443a93d794f72dc3626f"
I20260619 14:51:43.333405 20865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ed0e3a8c80a49e884beca712865f8e2" candidate_uuid: "4b630441204d4e5f8724d50b3a927a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd3a09b3febe464db009fe162a1d97b8"
I20260619 14:51:43.333487 20994 raft_consensus.cc:3037] T 1ed0e3a8c80a49e884beca712865f8e2 P d4d6ad23acd4443a93d794f72dc3626f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:43.333515 20865 raft_consensus.cc:3037] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:43.334491 20865 raft_consensus.cc:2454] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b630441204d4e5f8724d50b3a927a53 in term 1.
I20260619 14:51:43.334844 21077 leader_election.cc:302] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b630441204d4e5f8724d50b3a927a53, fd3a09b3febe464db009fe162a1d97b8; no voters:
I20260619 14:51:43.334965 20994 raft_consensus.cc:2454] T 1ed0e3a8c80a49e884beca712865f8e2 P d4d6ad23acd4443a93d794f72dc3626f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b630441204d4e5f8724d50b3a927a53 in term 1.
I20260619 14:51:43.334990 21220 raft_consensus.cc:2781] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:43.335177 21220 raft_consensus.cc:686] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 [term 1 LEADER]: Becoming Leader. State: Replica: 4b630441204d4e5f8724d50b3a927a53, State: Running, Role: LEADER
I20260619 14:51:43.335503 21220 consensus_queue.cc:237] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } }
I20260619 14:51:43.338241 20707 catalog_manager.cc:5495] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b630441204d4e5f8724d50b3a927a53 (127.19.34.67). New cstate: current_term: 1 leader_uuid: "4b630441204d4e5f8724d50b3a927a53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } health_report { overall_health: UNKNOWN } } }
I20260619 14:51:43.378574 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.375632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38592 (local address 127.19.34.67:44369)
0619 14:51:43.375742 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:43.375745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:43.375755 (+ 10us) server_negotiation.cc:407] Connection header received
0619 14:51:43.375913 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:43.375915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:43.375955 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:43.376024 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:43.376241 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.376706 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.377508 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.377681 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.377759 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.377991 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:43.378299 (+ 308us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:43.378371 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:43.378454 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 14:51:43.378494 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:51:43.383756 20865 raft_consensus.cc:1261] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8 [term 1 FOLLOWER]: Refusing update from remote peer 4b630441204d4e5f8724d50b3a927a53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:43.383803 20994 raft_consensus.cc:1261] T 1ed0e3a8c80a49e884beca712865f8e2 P d4d6ad23acd4443a93d794f72dc3626f [term 1 FOLLOWER]: Refusing update from remote peer 4b630441204d4e5f8724d50b3a927a53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:43.384485 21220 consensus_queue.cc:1035] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:51:43.384645 21224 consensus_queue.cc:1035] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:51:43.385041 21192 tablet.cc:2369] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:51:43.391448 21229 mvcc.cc:205] Tried to move back new op lower bound from 7298583361052061696 to 7298583360864862208. Current Snapshot: MvccSnapshot[applied={T|T < 7298583361052061696}]
I20260619 14:51:43.392320 21228 mvcc.cc:205] Tried to move back new op lower bound from 7298583361052061696 to 7298583360864862208. Current Snapshot: MvccSnapshot[applied={T|T < 7298583361052061696}]
I20260619 14:51:43.400409 20707 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:51:43.400520 20707 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:51:43.402465 20707 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8: Sending DeleteTablet for 3 replicas of tablet 1ed0e3a8c80a49e884beca712865f8e2
I20260619 14:51:43.403157 21125 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ed0e3a8c80a49e884beca712865f8e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:43 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:38580
I20260619 14:51:43.403180 20977 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ed0e3a8c80a49e884beca712865f8e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:43 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:46336
I20260619 14:51:43.403609 20845 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ed0e3a8c80a49e884beca712865f8e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:43 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:57956
I20260619 14:51:43.403774 21236 tablet_replica.cc:330] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53: stopping tablet replica
I20260619 14:51:43.404024 21238 tablet_replica.cc:330] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8: stopping tablet replica
I20260619 14:51:43.404227 21237 tablet_replica.cc:330] T 1ed0e3a8c80a49e884beca712865f8e2 P d4d6ad23acd4443a93d794f72dc3626f: stopping tablet replica
I20260619 14:51:43.404409 20707 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 14:51:43.404552 21237 raft_consensus.cc:2229] T 1ed0e3a8c80a49e884beca712865f8e2 P d4d6ad23acd4443a93d794f72dc3626f [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 14:51:43.404778 20707 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:51:43.404800 21237 raft_consensus.cc:2258] T 1ed0e3a8c80a49e884beca712865f8e2 P d4d6ad23acd4443a93d794f72dc3626f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:43.405069 21236 raft_consensus.cc:2229] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:51:43.405350 21236 raft_consensus.cc:2258] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:43.405532 21238 raft_consensus.cc:2229] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:51:43.405750 21238 raft_consensus.cc:2258] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:43.406970 21236 ts_tablet_manager.cc:1905] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:43.407124 21237 ts_tablet_manager.cc:1905] T 1ed0e3a8c80a49e884beca712865f8e2 P d4d6ad23acd4443a93d794f72dc3626f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:43.409638 20845 tablet_service.cc:1467] Processing CreateTablet for tablet 63811037af694de198dd3d48738aac43 (DEFAULT_TABLE table=test-table [id=7f5da069da1f4e2bb8116a6de50beb55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:43.409699 21237 ts_tablet_manager.cc:1918] T 1ed0e3a8c80a49e884beca712865f8e2 P d4d6ad23acd4443a93d794f72dc3626f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:51:43.409778 21237 log.cc:1196] T 1ed0e3a8c80a49e884beca712865f8e2 P d4d6ad23acd4443a93d794f72dc3626f: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-1/wal/wals/1ed0e3a8c80a49e884beca712865f8e2
I20260619 14:51:43.409873 20845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63811037af694de198dd3d48738aac43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:43.410070 21237 ts_tablet_manager.cc:1939] T 1ed0e3a8c80a49e884beca712865f8e2 P d4d6ad23acd4443a93d794f72dc3626f: Deleting consensus metadata
I20260619 14:51:43.410547 21236 ts_tablet_manager.cc:1918] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:51:43.410683 21236 log.cc:1196] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/1ed0e3a8c80a49e884beca712865f8e2
I20260619 14:51:43.411042 21236 ts_tablet_manager.cc:1939] T 1ed0e3a8c80a49e884beca712865f8e2 P 4b630441204d4e5f8724d50b3a927a53: Deleting consensus metadata
I20260619 14:51:43.411895 21238 ts_tablet_manager.cc:1905] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:43.412147 21214 tablet_bootstrap.cc:492] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8: Bootstrap starting.
I20260619 14:51:43.413060 21214 tablet_bootstrap.cc:654] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:43.413813 21238 ts_tablet_manager.cc:1918] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:51:43.413913 21238 log.cc:1196] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-0/wal/wals/1ed0e3a8c80a49e884beca712865f8e2
I20260619 14:51:43.414124 20694 catalog_manager.cc:4841] TS 4b630441204d4e5f8724d50b3a927a53 (127.19.34.67:44369): tablet 1ed0e3a8c80a49e884beca712865f8e2 (table test-table [id=3b80c698f6854353bca707561c2f6252]) successfully deleted
I20260619 14:51:43.414225 21238 ts_tablet_manager.cc:1939] T 1ed0e3a8c80a49e884beca712865f8e2 P fd3a09b3febe464db009fe162a1d97b8: Deleting consensus metadata
I20260619 14:51:43.414274 20694 catalog_manager.cc:4841] TS d4d6ad23acd4443a93d794f72dc3626f (127.19.34.66:34803): tablet 1ed0e3a8c80a49e884beca712865f8e2 (table test-table [id=3b80c698f6854353bca707561c2f6252]) successfully deleted
I20260619 14:51:43.414693 21214 tablet_bootstrap.cc:492] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8: No bootstrap required, opened a new log
I20260619 14:51:43.414724 21125 tablet_service.cc:1467] Processing CreateTablet for tablet 63811037af694de198dd3d48738aac43 (DEFAULT_TABLE table=test-table [id=7f5da069da1f4e2bb8116a6de50beb55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:43.414775 21214 ts_tablet_manager.cc:1397] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:51:43.414940 21125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63811037af694de198dd3d48738aac43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:43.415226 21214 raft_consensus.cc:348] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } }
I20260619 14:51:43.415326 21214 raft_consensus.cc:374] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:43.415357 21214 raft_consensus.cc:729] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3a09b3febe464db009fe162a1d97b8, State: Initialized, Role: FOLLOWER
I20260619 14:51:43.415401 20977 tablet_service.cc:1467] Processing CreateTablet for tablet 63811037af694de198dd3d48738aac43 (DEFAULT_TABLE table=test-table [id=7f5da069da1f4e2bb8116a6de50beb55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:43.415462 21214 consensus_queue.cc:260] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } }
I20260619 14:51:43.415609 20977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63811037af694de198dd3d48738aac43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:43.417256 21214 ts_tablet_manager.cc:1428] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:51:43.417117 21215 tablet_bootstrap.cc:492] T 63811037af694de198dd3d48738aac43 P 4b630441204d4e5f8724d50b3a927a53: Bootstrap starting.
I20260619 14:51:43.417748 21216 tablet_bootstrap.cc:492] T 63811037af694de198dd3d48738aac43 P d4d6ad23acd4443a93d794f72dc3626f: Bootstrap starting.
I20260619 14:51:43.418124 21215 tablet_bootstrap.cc:654] T 63811037af694de198dd3d48738aac43 P 4b630441204d4e5f8724d50b3a927a53: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:43.418689 21216 tablet_bootstrap.cc:654] T 63811037af694de198dd3d48738aac43 P d4d6ad23acd4443a93d794f72dc3626f: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:43.418905 20693 catalog_manager.cc:4841] TS fd3a09b3febe464db009fe162a1d97b8 (127.19.34.65:35021): tablet 1ed0e3a8c80a49e884beca712865f8e2 (table test-table [id=3b80c698f6854353bca707561c2f6252]) successfully deleted
I20260619 14:51:43.419694 21215 tablet_bootstrap.cc:492] T 63811037af694de198dd3d48738aac43 P 4b630441204d4e5f8724d50b3a927a53: No bootstrap required, opened a new log
I20260619 14:51:43.419726 21216 tablet_bootstrap.cc:492] T 63811037af694de198dd3d48738aac43 P d4d6ad23acd4443a93d794f72dc3626f: No bootstrap required, opened a new log
I20260619 14:51:43.419792 21215 ts_tablet_manager.cc:1397] T 63811037af694de198dd3d48738aac43 P 4b630441204d4e5f8724d50b3a927a53: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:51:43.419806 21216 ts_tablet_manager.cc:1397] T 63811037af694de198dd3d48738aac43 P d4d6ad23acd4443a93d794f72dc3626f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:43.420212 21215 raft_consensus.cc:348] T 63811037af694de198dd3d48738aac43 P 4b630441204d4e5f8724d50b3a927a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } }
I20260619 14:51:43.420200 21216 raft_consensus.cc:348] T 63811037af694de198dd3d48738aac43 P d4d6ad23acd4443a93d794f72dc3626f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } }
I20260619 14:51:43.420320 21216 raft_consensus.cc:374] T 63811037af694de198dd3d48738aac43 P d4d6ad23acd4443a93d794f72dc3626f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:43.420320 21215 raft_consensus.cc:374] T 63811037af694de198dd3d48738aac43 P 4b630441204d4e5f8724d50b3a927a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:43.420354 21216 raft_consensus.cc:729] T 63811037af694de198dd3d48738aac43 P d4d6ad23acd4443a93d794f72dc3626f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d6ad23acd4443a93d794f72dc3626f, State: Initialized, Role: FOLLOWER
I20260619 14:51:43.420354 21215 raft_consensus.cc:729] T 63811037af694de198dd3d48738aac43 P 4b630441204d4e5f8724d50b3a927a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b630441204d4e5f8724d50b3a927a53, State: Initialized, Role: FOLLOWER
I20260619 14:51:43.420467 21215 consensus_queue.cc:260] T 63811037af694de198dd3d48738aac43 P 4b630441204d4e5f8724d50b3a927a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } }
I20260619 14:51:43.420473 21216 consensus_queue.cc:260] T 63811037af694de198dd3d48738aac43 P d4d6ad23acd4443a93d794f72dc3626f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } }
I20260619 14:51:43.420710 21215 ts_tablet_manager.cc:1428] T 63811037af694de198dd3d48738aac43 P 4b630441204d4e5f8724d50b3a927a53: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:51:43.420724 21216 ts_tablet_manager.cc:1428] T 63811037af694de198dd3d48738aac43 P d4d6ad23acd4443a93d794f72dc3626f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260619 14:51:43.443037 21052 tablet.cc:2369] T 63811037af694de198dd3d48738aac43 P d4d6ad23acd4443a93d794f72dc3626f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:51:43.662532 21221 raft_consensus.cc:482] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:51:43.662647 21221 raft_consensus.cc:504] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } }
I20260619 14:51:43.663812 21221 leader_election.cc:288] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4d6ad23acd4443a93d794f72dc3626f (127.19.34.66:34803), 4b630441204d4e5f8724d50b3a927a53 (127.19.34.67:44369)
I20260619 14:51:43.667620 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.664461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:43481 (local address 127.19.34.67:44369)
0619 14:51:43.664596 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:43.664602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:43.664763 (+ 161us) server_negotiation.cc:407] Connection header received
0619 14:51:43.664889 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:43.664893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:43.664950 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:43.665019 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:43.665302 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.665837 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.666776 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.667146 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.667260 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.667309 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:43.667448 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 14:51:43.667511 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:51:43.667623 21213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.664117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:36745 (local address 127.19.34.66:34803)
0619 14:51:43.664269 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:43.664274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:43.664382 (+ 108us) server_negotiation.cc:407] Connection header received
0619 14:51:43.664537 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:43.664541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:43.664589 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:43.664655 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:43.664979 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.665595 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.666664 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.667068 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.667147 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.667183 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:43.667477 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 14:51:43.667536 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:51:43.667640 21242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.663979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:34803 (local address 127.19.34.65:36745)
0619 14:51:43.664274 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:43.664289 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:43.664419 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:43.664673 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:43.664675 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:43.664692 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:43.664868 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.664875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.665691 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.665694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.666539 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.666547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.667184 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.667189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.667326 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.667364 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 14:51:43.667448 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 14:51:43.667512 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":55,"thread_start_us":81,"threads_started":1}
I20260619 14:51:43.667651 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.664396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:44369 (local address 127.19.34.65:43481)
0619 14:51:43.664684 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:43.664695 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:43.664777 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:43.665051 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:43.665054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:43.665071 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:43.665201 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.665207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.665949 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.665952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.666667 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.666675 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.667258 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.667261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.667338 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.667366 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 14:51:43.667415 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 14:51:43.667496 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260619 14:51:43.668200 20994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63811037af694de198dd3d48738aac43" candidate_uuid: "fd3a09b3febe464db009fe162a1d97b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d6ad23acd4443a93d794f72dc3626f" is_pre_election: true
I20260619 14:51:43.668257 21145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63811037af694de198dd3d48738aac43" candidate_uuid: "fd3a09b3febe464db009fe162a1d97b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b630441204d4e5f8724d50b3a927a53" is_pre_election: true
I20260619 14:51:43.668354 20994 raft_consensus.cc:2454] T 63811037af694de198dd3d48738aac43 P d4d6ad23acd4443a93d794f72dc3626f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd3a09b3febe464db009fe162a1d97b8 in term 0.
I20260619 14:51:43.668484 21145 raft_consensus.cc:2454] T 63811037af694de198dd3d48738aac43 P 4b630441204d4e5f8724d50b3a927a53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd3a09b3febe464db009fe162a1d97b8 in term 0.
I20260619 14:51:43.668694 20798 leader_election.cc:302] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4d6ad23acd4443a93d794f72dc3626f, fd3a09b3febe464db009fe162a1d97b8; no voters:
I20260619 14:51:43.668918 21221 raft_consensus.cc:2781] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:51:43.668963 21221 raft_consensus.cc:482] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:51:43.668994 21221 raft_consensus.cc:3037] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:43.669896 21221 raft_consensus.cc:504] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } }
I20260619 14:51:43.670239 21221 leader_election.cc:288] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 [CANDIDATE]: Term 1 election: Requested vote from peers d4d6ad23acd4443a93d794f72dc3626f (127.19.34.66:34803), 4b630441204d4e5f8724d50b3a927a53 (127.19.34.67:44369)
I20260619 14:51:43.670472 20994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63811037af694de198dd3d48738aac43" candidate_uuid: "fd3a09b3febe464db009fe162a1d97b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d6ad23acd4443a93d794f72dc3626f"
I20260619 14:51:43.670567 20994 raft_consensus.cc:3037] T 63811037af694de198dd3d48738aac43 P d4d6ad23acd4443a93d794f72dc3626f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:43.670548 21145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63811037af694de198dd3d48738aac43" candidate_uuid: "fd3a09b3febe464db009fe162a1d97b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b630441204d4e5f8724d50b3a927a53"
I20260619 14:51:43.670661 21145 raft_consensus.cc:3037] T 63811037af694de198dd3d48738aac43 P 4b630441204d4e5f8724d50b3a927a53 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:43.671497 20994 raft_consensus.cc:2454] T 63811037af694de198dd3d48738aac43 P d4d6ad23acd4443a93d794f72dc3626f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd3a09b3febe464db009fe162a1d97b8 in term 1.
I20260619 14:51:43.671646 21145 raft_consensus.cc:2454] T 63811037af694de198dd3d48738aac43 P 4b630441204d4e5f8724d50b3a927a53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd3a09b3febe464db009fe162a1d97b8 in term 1.
I20260619 14:51:43.671759 20798 leader_election.cc:302] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4d6ad23acd4443a93d794f72dc3626f, fd3a09b3febe464db009fe162a1d97b8; no voters:
I20260619 14:51:43.671916 21221 raft_consensus.cc:2781] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:43.672123 21221 raft_consensus.cc:686] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 [term 1 LEADER]: Becoming Leader. State: Replica: fd3a09b3febe464db009fe162a1d97b8, State: Running, Role: LEADER
I20260619 14:51:43.672382 21221 consensus_queue.cc:237] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } }
I20260619 14:51:43.674369 20708 catalog_manager.cc:5495] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd3a09b3febe464db009fe162a1d97b8 (127.19.34.65). New cstate: current_term: 1 leader_uuid: "fd3a09b3febe464db009fe162a1d97b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } health_report { overall_health: UNKNOWN } } }
Jun 19 14:51:43 dist-test-slave-3qdh krb5kdc[20663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880702, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:43.741027 21252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.733823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51258 (local address 127.19.34.126:42433)
0619 14:51:43.734110 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:43.734116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:43.734131 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:51:43.734160 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:43.734162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:43.734196 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:43.734277 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:43.734841 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.735356 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.736079 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.736308 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.736379 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.738351 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:43.738353 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:43.738355 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:43.738371 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:43.739520 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:43.739911 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:43.739916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:43.739918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:43.739983 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:43.740212 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:43.740222 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:43.740224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:43.740423 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:43.740512 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:43.740758 (+ 246us) server_negotiation.cc:299] Negotiation successful
0619 14:51:43.740901 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":232,"thread_start_us":103,"threads_started":1}
I20260619 14:51:43.741029 21251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.733737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:42433 (local address 127.0.0.1:51258)
0619 14:51:43.733997 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:43.734011 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:43.734099 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:43.734307 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:43.734309 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:43.734655 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:43.734750 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.734755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.735457 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.735459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.735987 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.735993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.736385 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.736387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.736445 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.736990 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:43.737018 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:43.738230 (+ 1212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:43.739614 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:43.739623 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:43.739629 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:43.739826 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:43.740068 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:43.740071 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:43.740072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:43.740139 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:43.740518 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:43.740525 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:43.740616 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 14:51:43.740727 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 14:51:43.740851 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":81,"threads_started":1}
I20260619 14:51:43.744396 20708 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:51258:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 14:51:43.745029 20708 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:51:43.749332 20845 tablet_service.cc:1467] Processing CreateTablet for tablet d90339a9c15844218136010b45400f6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5f5ddc13291a4fd59b505fc511eb3c46]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:51:43.749413 20977 tablet_service.cc:1467] Processing CreateTablet for tablet d90339a9c15844218136010b45400f6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5f5ddc13291a4fd59b505fc511eb3c46]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:51:43.749709 20845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d90339a9c15844218136010b45400f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:43.749737 20977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d90339a9c15844218136010b45400f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:43.749619 21125 tablet_service.cc:1467] Processing CreateTablet for tablet d90339a9c15844218136010b45400f6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5f5ddc13291a4fd59b505fc511eb3c46]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:51:43.749917 21125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d90339a9c15844218136010b45400f6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:43.752375 21216 tablet_bootstrap.cc:492] T d90339a9c15844218136010b45400f6e P d4d6ad23acd4443a93d794f72dc3626f: Bootstrap starting.
I20260619 14:51:43.752385 21214 tablet_bootstrap.cc:492] T d90339a9c15844218136010b45400f6e P fd3a09b3febe464db009fe162a1d97b8: Bootstrap starting.
I20260619 14:51:43.752390 21215 tablet_bootstrap.cc:492] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53: Bootstrap starting.
I20260619 14:51:43.753226 21216 tablet_bootstrap.cc:654] T d90339a9c15844218136010b45400f6e P d4d6ad23acd4443a93d794f72dc3626f: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:43.753243 21214 tablet_bootstrap.cc:654] T d90339a9c15844218136010b45400f6e P fd3a09b3febe464db009fe162a1d97b8: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:43.753243 21215 tablet_bootstrap.cc:654] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:43.754542 21215 tablet_bootstrap.cc:492] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53: No bootstrap required, opened a new log
I20260619 14:51:43.754634 21215 ts_tablet_manager.cc:1397] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:51:43.754995 21214 tablet_bootstrap.cc:492] T d90339a9c15844218136010b45400f6e P fd3a09b3febe464db009fe162a1d97b8: No bootstrap required, opened a new log
I20260619 14:51:43.754982 21215 raft_consensus.cc:348] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } }
I20260619 14:51:43.755056 21215 raft_consensus.cc:374] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:43.755054 21214 ts_tablet_manager.cc:1397] T d90339a9c15844218136010b45400f6e P fd3a09b3febe464db009fe162a1d97b8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 14:51:43.755079 21215 raft_consensus.cc:729] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b630441204d4e5f8724d50b3a927a53, State: Initialized, Role: FOLLOWER
I20260619 14:51:43.755206 21215 consensus_queue.cc:260] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } }
I20260619 14:51:43.755421 21214 raft_consensus.cc:348] T d90339a9c15844218136010b45400f6e P fd3a09b3febe464db009fe162a1d97b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } }
I20260619 14:51:43.755429 21220 tablet_replica.cc:439] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } } }
I20260619 14:51:43.755517 21214 raft_consensus.cc:374] T d90339a9c15844218136010b45400f6e P fd3a09b3febe464db009fe162a1d97b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:43.755541 21214 raft_consensus.cc:729] T d90339a9c15844218136010b45400f6e P fd3a09b3febe464db009fe162a1d97b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3a09b3febe464db009fe162a1d97b8, State: Initialized, Role: FOLLOWER
I20260619 14:51:43.755558 21220 tablet_replica.cc:442] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:51:43.755654 21214 consensus_queue.cc:260] T d90339a9c15844218136010b45400f6e P fd3a09b3febe464db009fe162a1d97b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } }
I20260619 14:51:43.755816 21221 tablet_replica.cc:439] T d90339a9c15844218136010b45400f6e P fd3a09b3febe464db009fe162a1d97b8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } } }
I20260619 14:51:43.755873 21214 ts_tablet_manager.cc:1428] T d90339a9c15844218136010b45400f6e P fd3a09b3febe464db009fe162a1d97b8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:51:43.755903 21221 tablet_replica.cc:442] T d90339a9c15844218136010b45400f6e P fd3a09b3febe464db009fe162a1d97b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:51:43.755497 21215 ts_tablet_manager.cc:1428] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:51:43.756626 21216 tablet_bootstrap.cc:492] T d90339a9c15844218136010b45400f6e P d4d6ad23acd4443a93d794f72dc3626f: No bootstrap required, opened a new log
I20260619 14:51:43.756723 21216 ts_tablet_manager.cc:1397] T d90339a9c15844218136010b45400f6e P d4d6ad23acd4443a93d794f72dc3626f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 14:51:43.757099 21216 raft_consensus.cc:348] T d90339a9c15844218136010b45400f6e P d4d6ad23acd4443a93d794f72dc3626f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } }
I20260619 14:51:43.757196 21216 raft_consensus.cc:374] T d90339a9c15844218136010b45400f6e P d4d6ad23acd4443a93d794f72dc3626f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:43.757249 21216 raft_consensus.cc:729] T d90339a9c15844218136010b45400f6e P d4d6ad23acd4443a93d794f72dc3626f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d6ad23acd4443a93d794f72dc3626f, State: Initialized, Role: FOLLOWER
I20260619 14:51:43.757359 21216 consensus_queue.cc:260] T d90339a9c15844218136010b45400f6e P d4d6ad23acd4443a93d794f72dc3626f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } }
I20260619 14:51:43.757611 21216 ts_tablet_manager.cc:1428] T d90339a9c15844218136010b45400f6e P d4d6ad23acd4443a93d794f72dc3626f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 14:51:43.757570 21222 tablet_replica.cc:439] T d90339a9c15844218136010b45400f6e P d4d6ad23acd4443a93d794f72dc3626f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } } }
I20260619 14:51:43.757671 21222 tablet_replica.cc:442] T d90339a9c15844218136010b45400f6e P d4d6ad23acd4443a93d794f72dc3626f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:51:43.913837 21220 raft_consensus.cc:482] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:51:43.913944 21220 raft_consensus.cc:504] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } }
I20260619 14:51:43.914275 21220 leader_election.cc:288] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd3a09b3febe464db009fe162a1d97b8 (127.19.34.65:35021), d4d6ad23acd4443a93d794f72dc3626f (127.19.34.66:34803)
I20260619 14:51:43.914615 20994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90339a9c15844218136010b45400f6e" candidate_uuid: "4b630441204d4e5f8724d50b3a927a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d6ad23acd4443a93d794f72dc3626f" is_pre_election: true
I20260619 14:51:43.914615 20865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90339a9c15844218136010b45400f6e" candidate_uuid: "4b630441204d4e5f8724d50b3a927a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd3a09b3febe464db009fe162a1d97b8" is_pre_election: true
I20260619 14:51:43.914796 20994 raft_consensus.cc:2454] T d90339a9c15844218136010b45400f6e P d4d6ad23acd4443a93d794f72dc3626f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b630441204d4e5f8724d50b3a927a53 in term 0.
I20260619 14:51:43.914808 20865 raft_consensus.cc:2454] T d90339a9c15844218136010b45400f6e P fd3a09b3febe464db009fe162a1d97b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b630441204d4e5f8724d50b3a927a53 in term 0.
I20260619 14:51:43.915102 21077 leader_election.cc:302] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b630441204d4e5f8724d50b3a927a53, fd3a09b3febe464db009fe162a1d97b8; no voters:
I20260619 14:51:43.915306 21220 raft_consensus.cc:2781] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:51:43.915381 21220 raft_consensus.cc:482] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:51:43.915426 21220 raft_consensus.cc:3037] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:43.916361 21220 raft_consensus.cc:504] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } }
I20260619 14:51:43.916615 21220 leader_election.cc:288] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53 [CANDIDATE]: Term 1 election: Requested vote from peers fd3a09b3febe464db009fe162a1d97b8 (127.19.34.65:35021), d4d6ad23acd4443a93d794f72dc3626f (127.19.34.66:34803)
I20260619 14:51:43.916934 20865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90339a9c15844218136010b45400f6e" candidate_uuid: "4b630441204d4e5f8724d50b3a927a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd3a09b3febe464db009fe162a1d97b8"
I20260619 14:51:43.916949 20994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d90339a9c15844218136010b45400f6e" candidate_uuid: "4b630441204d4e5f8724d50b3a927a53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4d6ad23acd4443a93d794f72dc3626f"
I20260619 14:51:43.917042 20994 raft_consensus.cc:3037] T d90339a9c15844218136010b45400f6e P d4d6ad23acd4443a93d794f72dc3626f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:43.917042 20865 raft_consensus.cc:3037] T d90339a9c15844218136010b45400f6e P fd3a09b3febe464db009fe162a1d97b8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:43.917999 20865 raft_consensus.cc:2454] T d90339a9c15844218136010b45400f6e P fd3a09b3febe464db009fe162a1d97b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b630441204d4e5f8724d50b3a927a53 in term 1.
I20260619 14:51:43.917999 20994 raft_consensus.cc:2454] T d90339a9c15844218136010b45400f6e P d4d6ad23acd4443a93d794f72dc3626f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b630441204d4e5f8724d50b3a927a53 in term 1.
I20260619 14:51:43.918220 21078 leader_election.cc:302] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b630441204d4e5f8724d50b3a927a53, d4d6ad23acd4443a93d794f72dc3626f; no voters:
I20260619 14:51:43.918388 21220 raft_consensus.cc:2781] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:43.918541 21220 raft_consensus.cc:686] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53 [term 1 LEADER]: Becoming Leader. State: Replica: 4b630441204d4e5f8724d50b3a927a53, State: Running, Role: LEADER
I20260619 14:51:43.918681 21220 consensus_queue.cc:237] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } }
I20260619 14:51:43.919489 21256 tablet_replica.cc:439] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53: TxnStatusTablet state changed. Reason: New leader 4b630441204d4e5f8724d50b3a927a53. Latest consensus state: current_term: 1 leader_uuid: "4b630441204d4e5f8724d50b3a927a53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } } }
I20260619 14:51:43.919593 21256 tablet_replica.cc:442] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:51:43.919803 21258 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 14:51:43.919987 21258 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 14:51:43.920526 20707 catalog_manager.cc:5495] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b630441204d4e5f8724d50b3a927a53 (127.19.34.67). New cstate: current_term: 1 leader_uuid: "4b630441204d4e5f8724d50b3a927a53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } health_report { overall_health: HEALTHY } } }
I20260619 14:51:43.945590 21251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.942385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:44369 (local address 127.0.0.1:38606)
0619 14:51:43.942512 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:43.942523 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:43.942643 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:43.942992 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:43.942994 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:43.943000 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:51:43.943080 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.943084 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.943807 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.943812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.944512 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.944522 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.944960 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.944963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.945046 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.945081 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:51:43.945337 (+ 256us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:51:43.945351 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 14:51:43.945406 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 14:51:43.945484 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 14:51:43.945590 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.942295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38606 (local address 127.19.34.67:44369)
0619 14:51:43.942412 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:43.942418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:43.942583 (+ 165us) server_negotiation.cc:407] Connection header received
0619 14:51:43.942764 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:43.942768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:43.942817 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:43.942885 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:43.943197 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.943698 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.944656 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.944865 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.944959 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.945198 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:43.945267 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:43.945334 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:43.945442 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 14:51:43.945484 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:51:43.947660 20994 raft_consensus.cc:1261] T d90339a9c15844218136010b45400f6e P d4d6ad23acd4443a93d794f72dc3626f [term 1 FOLLOWER]: Refusing update from remote peer 4b630441204d4e5f8724d50b3a927a53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:43.947893 20865 raft_consensus.cc:1261] T d90339a9c15844218136010b45400f6e P fd3a09b3febe464db009fe162a1d97b8 [term 1 FOLLOWER]: Refusing update from remote peer 4b630441204d4e5f8724d50b3a927a53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:43.948078 21256 consensus_queue.cc:1035] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:51:43.948323 21220 consensus_queue.cc:1035] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:51:43.949344 21222 tablet_replica.cc:439] T d90339a9c15844218136010b45400f6e P d4d6ad23acd4443a93d794f72dc3626f: TxnStatusTablet state changed. Reason: New leader 4b630441204d4e5f8724d50b3a927a53. Latest consensus state: current_term: 1 leader_uuid: "4b630441204d4e5f8724d50b3a927a53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } } }
I20260619 14:51:43.949473 21222 tablet_replica.cc:442] T d90339a9c15844218136010b45400f6e P d4d6ad23acd4443a93d794f72dc3626f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:51:43.949911 21221 tablet_replica.cc:439] T d90339a9c15844218136010b45400f6e P fd3a09b3febe464db009fe162a1d97b8: TxnStatusTablet state changed. Reason: New leader 4b630441204d4e5f8724d50b3a927a53. Latest consensus state: current_term: 1 leader_uuid: "4b630441204d4e5f8724d50b3a927a53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } } }
I20260619 14:51:43.950016 21221 tablet_replica.cc:442] T d90339a9c15844218136010b45400f6e P fd3a09b3febe464db009fe162a1d97b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:51:43.951149 21220 tablet_replica.cc:439] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b630441204d4e5f8724d50b3a927a53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } } }
I20260619 14:51:43.951241 21220 tablet_replica.cc:442] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:51:43.951963 21221 tablet_replica.cc:439] T d90339a9c15844218136010b45400f6e P fd3a09b3febe464db009fe162a1d97b8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b630441204d4e5f8724d50b3a927a53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } } }
I20260619 14:51:43.952095 21221 tablet_replica.cc:442] T d90339a9c15844218136010b45400f6e P fd3a09b3febe464db009fe162a1d97b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:51:43.961324 21256 tablet_replica.cc:439] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b630441204d4e5f8724d50b3a927a53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } } }
I20260619 14:51:43.961422 21256 tablet_replica.cc:442] T d90339a9c15844218136010b45400f6e P 4b630441204d4e5f8724d50b3a927a53: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:51:43.962136 21222 tablet_replica.cc:439] T d90339a9c15844218136010b45400f6e P d4d6ad23acd4443a93d794f72dc3626f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b630441204d4e5f8724d50b3a927a53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3a09b3febe464db009fe162a1d97b8" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 35021 } } peers { permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 } } peers { permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 } } }
I20260619 14:51:43.962263 21222 tablet_replica.cc:442] T d90339a9c15844218136010b45400f6e P d4d6ad23acd4443a93d794f72dc3626f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:51:43.965560 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.956584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57960 (local address 127.19.34.65:35021)
0619 14:51:43.961373 (+ 4789us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:43.961377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:43.961389 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:51:43.961453 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:43.961455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:43.961514 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:43.961609 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:43.962054 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.962803 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.964018 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.964294 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.964438 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.964743 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:43.965163 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:43.965296 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:43.965358 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 14:51:43.965413 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4729,"thread_start_us":61,"threads_started":1}
I20260619 14:51:43.976248 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.972567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:44369 (local address 127.19.34.65:42373)
0619 14:51:43.972858 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:43.972871 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:43.972974 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:43.973216 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:43.973218 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:43.973223 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:51:43.973319 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.973323 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.974273 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.974276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.974869 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:43.974901 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.975536 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.975542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:43.975640 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.975687 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:51:43.976001 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:51:43.976006 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:51:43.976075 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 14:51:43.976134 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":113,"threads_started":1}
I20260619 14:51:43.976332 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.972665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:42373 (local address 127.19.34.67:44369)
0619 14:51:43.972784 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:43.972789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:43.972963 (+ 174us) server_negotiation.cc:407] Connection header received
0619 14:51:43.973048 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:43.973051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:43.973107 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:43.973202 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:43.973416 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.974155 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.975017 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:43.975368 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:43.975475 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:43.975831 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:43.975910 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:43.975990 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:43.976109 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 14:51:43.976176 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:51:43.983129 20994 raft_consensus.cc:1261] T 63811037af694de198dd3d48738aac43 P d4d6ad23acd4443a93d794f72dc3626f [term 1 FOLLOWER]: Refusing update from remote peer fd3a09b3febe464db009fe162a1d97b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:43.983318 21145 raft_consensus.cc:1261] T 63811037af694de198dd3d48738aac43 P 4b630441204d4e5f8724d50b3a927a53 [term 1 FOLLOWER]: Refusing update from remote peer fd3a09b3febe464db009fe162a1d97b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:43.983544 21221 consensus_queue.cc:1035] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4d6ad23acd4443a93d794f72dc3626f" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:51:43.983788 21244 consensus_queue.cc:1035] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b630441204d4e5f8724d50b3a927a53" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:51:43.988368 21261 mvcc.cc:205] Tried to move back new op lower bound from 7298583363510460416 to 7298583362244210688. Current Snapshot: MvccSnapshot[applied={T|T < 7298583363510460416}]
I20260619 14:51:44.002977 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.999786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:34527 (local address 127.19.34.65:35021)
0619 14:51:43.999971 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:43.999974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:44.000080 (+ 106us) server_negotiation.cc:407] Connection header received
0619 14:51:44.000203 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:44.000205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:44.000235 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:44.000284 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:44.000476 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:44.000964 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:44.001994 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:44.002167 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:44.002239 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:44.002521 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:44.002573 (+ 52us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:44.002623 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:44.002819 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 14:51:44.002866 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260619 14:51:44.003226 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:43.999662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:35021 (local address 127.19.34.67:34527)
0619 14:51:44.000022 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:44.000035 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:44.000137 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:44.000313 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:44.000316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:44.000322 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:51:44.000409 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:44.000414 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:44.001066 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:44.001069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:44.001902 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:44.001912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:44.002263 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:44.002267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:44.002353 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:44.002418 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:51:44.002704 (+ 286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:51:44.002713 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 14:51:44.002799 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 14:51:44.002883 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":120,"threads_started":1}
I20260619 14:51:44.023142 20707 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:51:44.023285 20707 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:51:44.025118 20707 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8f103ab00ac4cec847d60ba0e167ca8: Sending DeleteTablet for 3 replicas of tablet 63811037af694de198dd3d48738aac43
I20260619 14:51:44.025589 20845 tablet_service.cc:1514] Processing DeleteTablet for tablet 63811037af694de198dd3d48738aac43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:44 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:57956
I20260619 14:51:44.025908 21283 tablet_replica.cc:330] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8: stopping tablet replica
I20260619 14:51:44.025895 21125 tablet_service.cc:1514] Processing DeleteTablet for tablet 63811037af694de198dd3d48738aac43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:44 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:38580
I20260619 14:51:44.026010 21283 raft_consensus.cc:2229] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:51:44.026271 21283 raft_consensus.cc:2258] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:44.026255 20977 tablet_service.cc:1514] Processing DeleteTablet for tablet 63811037af694de198dd3d48738aac43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:44 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:46336
I20260619 14:51:44.026830 21283 ts_tablet_manager.cc:1905] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:44.027597 21284 tablet_replica.cc:330] T 63811037af694de198dd3d48738aac43 P 4b630441204d4e5f8724d50b3a927a53: stopping tablet replica
I20260619 14:51:44.027765 21285 tablet_replica.cc:330] T 63811037af694de198dd3d48738aac43 P d4d6ad23acd4443a93d794f72dc3626f: stopping tablet replica
I20260619 14:51:44.028395 21285 raft_consensus.cc:2229] T 63811037af694de198dd3d48738aac43 P d4d6ad23acd4443a93d794f72dc3626f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:51:44.028565 21284 raft_consensus.cc:2229] T 63811037af694de198dd3d48738aac43 P 4b630441204d4e5f8724d50b3a927a53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:51:44.029017 21284 raft_consensus.cc:2258] T 63811037af694de198dd3d48738aac43 P 4b630441204d4e5f8724d50b3a927a53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:44.029188 21285 raft_consensus.cc:2258] T 63811037af694de198dd3d48738aac43 P d4d6ad23acd4443a93d794f72dc3626f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:44.029767 21283 ts_tablet_manager.cc:1918] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:51:44.029839 21283 log.cc:1196] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-0/wal/wals/63811037af694de198dd3d48738aac43
I20260619 14:51:44.030041 21284 ts_tablet_manager.cc:1905] T 63811037af694de198dd3d48738aac43 P 4b630441204d4e5f8724d50b3a927a53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:44.030112 21283 ts_tablet_manager.cc:1939] T 63811037af694de198dd3d48738aac43 P fd3a09b3febe464db009fe162a1d97b8: Deleting consensus metadata
I20260619 14:51:44.030704 20693 catalog_manager.cc:4841] TS fd3a09b3febe464db009fe162a1d97b8 (127.19.34.65:35021): tablet 63811037af694de198dd3d48738aac43 (table test-table [id=7f5da069da1f4e2bb8116a6de50beb55]) successfully deleted
I20260619 14:51:44.031949 21285 ts_tablet_manager.cc:1905] T 63811037af694de198dd3d48738aac43 P d4d6ad23acd4443a93d794f72dc3626f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:44.032622 21284 ts_tablet_manager.cc:1918] T 63811037af694de198dd3d48738aac43 P 4b630441204d4e5f8724d50b3a927a53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:51:44.032696 21284 log.cc:1196] T 63811037af694de198dd3d48738aac43 P 4b630441204d4e5f8724d50b3a927a53: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/63811037af694de198dd3d48738aac43
I20260619 14:51:44.032968 21284 ts_tablet_manager.cc:1939] T 63811037af694de198dd3d48738aac43 P 4b630441204d4e5f8724d50b3a927a53: Deleting consensus metadata
I20260619 14:51:44.033598 20694 catalog_manager.cc:4841] TS 4b630441204d4e5f8724d50b3a927a53 (127.19.34.67:44369): tablet 63811037af694de198dd3d48738aac43 (table test-table [id=7f5da069da1f4e2bb8116a6de50beb55]) successfully deleted
I20260619 14:51:44.034941 21285 ts_tablet_manager.cc:1918] T 63811037af694de198dd3d48738aac43 P d4d6ad23acd4443a93d794f72dc3626f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:51:44.035029 21285 log.cc:1196] T 63811037af694de198dd3d48738aac43 P d4d6ad23acd4443a93d794f72dc3626f: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880697016847-19593-0/minicluster-data/ts-1/wal/wals/63811037af694de198dd3d48738aac43
I20260619 14:51:44.035317 21285 ts_tablet_manager.cc:1939] T 63811037af694de198dd3d48738aac43 P d4d6ad23acd4443a93d794f72dc3626f: Deleting consensus metadata
I20260619 14:51:44.035952 20694 catalog_manager.cc:4841] TS d4d6ad23acd4443a93d794f72dc3626f (127.19.34.66:34803): tablet 63811037af694de198dd3d48738aac43 (table test-table [id=7f5da069da1f4e2bb8116a6de50beb55]) successfully deleted
Jun 19 14:51:44 dist-test-slave-3qdh krb5kdc[20663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880703, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.34.65@KRBTEST.COM
I20260619 14:51:44.047287 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:44.028126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57966 (local address 127.19.34.65:35021)
0619 14:51:44.028251 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:44.028260 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:44.036560 (+ 8300us) server_negotiation.cc:407] Connection header received
0619 14:51:44.036654 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:44.036657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:44.036700 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:44.036782 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:44.037550 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:44.038310 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:44.039456 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:44.039758 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:44.039877 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:44.043418 (+ 3541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:44.043428 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:44.043442 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:44.043466 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:44.045432 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:44.045948 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:44.045953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:44.045955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:44.046039 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:44.046327 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:44.046330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:44.046331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:44.046560 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:44.046757 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:44.046955 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 14:51:44.047060 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 14:51:44.047675 20905 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57966
I20260619 14:51:44.058506 21252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:44.049337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51260 (local address 127.19.34.126:42433)
0619 14:51:44.049448 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:44.049453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:44.049580 (+ 127us) server_negotiation.cc:407] Connection header received
0619 14:51:44.049776 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:44.049779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:44.049828 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:44.049881 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:44.050776 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:44.051467 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:44.052589 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:44.052958 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:44.053047 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:44.054813 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:44.054817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:44.054820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:44.054846 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:44.056472 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:44.056971 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:44.056975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:44.056977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:44.057068 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:44.057459 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:44.057463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:44.057465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:44.057703 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:44.057815 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:44.058209 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 14:51:44.058341 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 14:51:44.059330 20707 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51260
I20260619 14:51:44.062021 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 20780
I20260619 14:51:44.068579 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 20920
I20260619 14:51:44.074308 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 21060
I20260619 14:51:44.080682 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 20676
2026-06-19T14:51:44Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2827 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:44 dist-test-slave-3qdh krb5kdc[21292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:44 dist-test-slave-3qdh krb5kdc[21292](info): set up 2 sockets
Jun 19 14:51:44 dist-test-slave-3qdh krb5kdc[21292](info): commencing operation
krb5kdc: starting...
W20260619 14:51:45.079922 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:51:45 dist-test-slave-3qdh krb5kdc[21292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:51:45.185003 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:33109
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:33109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:45.283263 21305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:45.283514 21305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:45.283584 21305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:45.287335 21305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:45.287410 21305 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 14:51:45.287439 21305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:45.287474 21305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:45.287503 21305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:45.287533 21305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:45.291810 21305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:33109
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:33109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:45.292706 21305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:45.293565 21305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:45.300756 21310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:45.300828 21305 server_base.cc:1034] running on GCE node
W20260619 14:51:45.300760 21311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:45.300761 21313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:45.301506 21305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:45.302403 21305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:45.303578 21305 hybrid_clock.cc:648] HybridClock initialized: now 1781880705303554 us; error 45 us; skew 500 ppm
Jun 19 14:51:45 dist-test-slave-3qdh krb5kdc[21292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880705, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:45.305910 21305 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:51:45.306661 21305 webserver.cc:460] Webserver started at http://127.19.34.126:37711/ using document root <none> and password file <none>
I20260619 14:51:45.307067 21305 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:45.307116 21305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:45.307256 21305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:45.308945 21305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "58d8ade3b66a41e5bb311f4f347e30de"
format_stamp: "Formatted at 2026-06-19 14:51:45 on dist-test-slave-3qdh"
server_key: "158d3b09790342049a53cd09ff7c20c7"
server_key_iv: "2ea7a5e287de709224d17602ff9577fc"
server_key_version: "encryptionkey@0"
I20260619 14:51:45.309423 21305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "58d8ade3b66a41e5bb311f4f347e30de"
format_stamp: "Formatted at 2026-06-19 14:51:45 on dist-test-slave-3qdh"
server_key: "158d3b09790342049a53cd09ff7c20c7"
server_key_iv: "2ea7a5e287de709224d17602ff9577fc"
server_key_version: "encryptionkey@0"
I20260619 14:51:45.312541 21305 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:51:45.314581 21320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:45.315629 21305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:45.315761 21305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "58d8ade3b66a41e5bb311f4f347e30de"
format_stamp: "Formatted at 2026-06-19 14:51:45 on dist-test-slave-3qdh"
server_key: "158d3b09790342049a53cd09ff7c20c7"
server_key_iv: "2ea7a5e287de709224d17602ff9577fc"
server_key_version: "encryptionkey@0"
I20260619 14:51:45.315851 21305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:45.336774 21305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:45.339144 21305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:45.339319 21305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:45.346138 21305 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:33109
I20260619 14:51:45.346163 21382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:33109 every 8 connection(s)
I20260619 14:51:45.346846 21305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:45.349112 21383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:45.349299 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 21305
I20260619 14:51:45.349388 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:51:45.349642 19593 external_mini_cluster.cc:1428] Setting key 3fa711235329682eb079e723d5560aed
I20260619 14:51:45.354691 21383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de: Bootstrap starting.
I20260619 14:51:45.356709 21383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:45.357336 21383 log.cc:824] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:51:45 dist-test-slave-3qdh krb5kdc[21292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:45.358932 21383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de: No bootstrap required, opened a new log
I20260619 14:51:45.361632 21383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d8ade3b66a41e5bb311f4f347e30de" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33109 } }
I20260619 14:51:45.361863 21383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:45.361927 21383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d8ade3b66a41e5bb311f4f347e30de, State: Initialized, Role: FOLLOWER
I20260619 14:51:45.362340 21383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d8ade3b66a41e5bb311f4f347e30de" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33109 } }
I20260619 14:51:45.362485 21383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:45.362568 21383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:45.362681 21383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:45.362825 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:45.350888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49362 (local address 127.19.34.126:33109)
0619 14:51:45.351276 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:45.351290 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:45.351320 (+ 30us) server_negotiation.cc:407] Connection header received
0619 14:51:45.351952 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:45.351966 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:45.352232 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:45.352568 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:45.353301 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:45.354249 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:45.355132 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:45.355581 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:45.355756 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:45.357983 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:45.357988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:45.357998 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:45.358021 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:45.360059 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:45.360567 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:45.360572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:45.360584 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:45.360695 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:45.360993 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:45.360997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:45.360998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:45.361320 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:45.361535 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:45.361773 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 14:51:45.362059 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":118,"threads_started":1}
I20260619 14:51:45.363564 21383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d8ade3b66a41e5bb311f4f347e30de" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33109 } }
I20260619 14:51:45.363879 21383 leader_election.cc:302] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58d8ade3b66a41e5bb311f4f347e30de; no voters:
I20260619 14:51:45.364199 21383 leader_election.cc:288] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:45.364259 21388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:45.364491 21388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de [term 1 LEADER]: Becoming Leader. State: Replica: 58d8ade3b66a41e5bb311f4f347e30de, State: Running, Role: LEADER
I20260619 14:51:45.364743 21388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d8ade3b66a41e5bb311f4f347e30de" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33109 } }
I20260619 14:51:45.365875 21383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:45.366421 21389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58d8ade3b66a41e5bb311f4f347e30de. Latest consensus state: current_term: 1 leader_uuid: "58d8ade3b66a41e5bb311f4f347e30de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d8ade3b66a41e5bb311f4f347e30de" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33109 } } }
I20260619 14:51:45.366546 21389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:45.366899 21390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58d8ade3b66a41e5bb311f4f347e30de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58d8ade3b66a41e5bb311f4f347e30de" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33109 } } }
I20260619 14:51:45.367003 21390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de [sys.catalog]: This master's current role is: LEADER
W20260619 14:51:45.370808 21403 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:51:45.370888 21403 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:51:45.370955 21394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:45.371848 21394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:45.377308 21394 catalog_manager.cc:1347] Generated new cluster ID: 2f6a8dda6ce64ced8dd2d6cf8cd82c2a
I20260619 14:51:45.377388 21394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:45.403313 21394 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:45.404114 21394 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:45.410878 21394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de: Generated new TSK 0
I20260619 14:51:45.411425 21394 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.34.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
I20260619 14:51:45.439347 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:33109
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:45.558123 21409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:45.558333 21409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:45.558393 21409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:45.561906 21409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:45.561996 21409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:45.562072 21409 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 14:51:45.562114 21409 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:51:45.562176 21409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:51:45.567672 21409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.34.126:33109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:45.568792 21409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:45.569878 21409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:45.580308 21415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:45.580415 21414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:45.580497 21409 server_base.cc:1034] running on GCE node
W20260619 14:51:45.580700 21417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:45.581065 21409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:45.581624 21409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:45.582806 21409 hybrid_clock.cc:648] HybridClock initialized: now 1781880705582784 us; error 40 us; skew 500 ppm
Jun 19 14:51:45 dist-test-slave-3qdh krb5kdc[21292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880705, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:45.585413 21409 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:51:45.586184 21409 webserver.cc:460] Webserver started at http://127.19.34.65:40541/ using document root <none> and password file <none>
I20260619 14:51:45.586649 21409 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:45.586701 21409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:45.586901 21409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:45.589184 21409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "4d56944bba9847c4b319a72f051808ad"
format_stamp: "Formatted at 2026-06-19 14:51:45 on dist-test-slave-3qdh"
server_key: "429dc541bb321cfdc71ebec09fc94286"
server_key_iv: "e05d3383bd37cefdb6710eb57744507b"
server_key_version: "encryptionkey@0"
I20260619 14:51:45.589751 21409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "4d56944bba9847c4b319a72f051808ad"
format_stamp: "Formatted at 2026-06-19 14:51:45 on dist-test-slave-3qdh"
server_key: "429dc541bb321cfdc71ebec09fc94286"
server_key_iv: "e05d3383bd37cefdb6710eb57744507b"
server_key_version: "encryptionkey@0"
I20260619 14:51:45.593266 21409 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:51:45.595450 21424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:45.596721 21409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:51:45.596838 21409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "4d56944bba9847c4b319a72f051808ad"
format_stamp: "Formatted at 2026-06-19 14:51:45 on dist-test-slave-3qdh"
server_key: "429dc541bb321cfdc71ebec09fc94286"
server_key_iv: "e05d3383bd37cefdb6710eb57744507b"
server_key_version: "encryptionkey@0"
I20260619 14:51:45.596949 21409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:45.617656 21409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:45.620384 21409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:45.620500 21409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:45.621737 21409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:45.621789 21409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:45.621846 21409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:45.621878 21409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:51:45 dist-test-slave-3qdh krb5kdc[21292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880705, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:45.632506 21409 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:33029
I20260619 14:51:45.632586 21539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:33029 every 8 connection(s)
I20260619 14:51:45.633481 21409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:51:45.634953 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 21409
I20260619 14:51:45.635208 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:51:45.635504 19593 external_mini_cluster.cc:1428] Setting key 68b7ef6b911836d7ed3494eab5e368ac
I20260619 14:51:45.635502 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:45.622547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:54653 (local address 127.19.34.126:33109)
0619 14:51:45.622683 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:45.622686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:45.623589 (+ 903us) server_negotiation.cc:407] Connection header received
0619 14:51:45.624824 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:45.624828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:45.624893 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:45.624990 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:45.626356 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:45.627241 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:45.628730 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:45.628981 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:45.629125 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:45.631639 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:45.631650 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:45.631653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:45.631680 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:45.633377 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:45.633949 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:45.633952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:45.633954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:45.634041 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:45.634442 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:45.634445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:45.634446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:45.634621 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:45.634707 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:45.635213 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 14:51:45.635334 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:51:45.636099 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:45.622985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:33109 (local address 127.19.34.65:54653)
0619 14:51:45.623448 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:45.623486 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:45.624508 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:45.625190 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:45.625203 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:45.625670 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:45.626180 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:45.626193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:45.627374 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:45.627378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:45.628562 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:45.628574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:45.629179 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:45.629183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:45.629343 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:45.629963 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:45.629979 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:45.631496 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:45.633509 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:45.633513 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:45.633522 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:45.633817 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:45.634170 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:45.634173 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:45.634175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:45.634356 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:45.634717 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:45.634721 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:45.634892 (+ 171us) client_negotiation.cc:769] Sending connection context
0619 14:51:45.635053 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 14:51:45.635326 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":135,"threads_started":1}
I20260619 14:51:45.637425 21540 heartbeater.cc:347] Connected to a master server at 127.19.34.126:33109
I20260619 14:51:45.637552 21540 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:45.638015 21540 heartbeater.cc:511] Master 127.19.34.126:33109 requested a full tablet report, sending...
I20260619 14:51:45.639320 21336 ts_manager.cc:194] Registered new tserver with Master: 4d56944bba9847c4b319a72f051808ad (127.19.34.65:33029)
I20260619 14:51:45.640751 21336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:54653
WARNING: no policy specified for kudu/127.19.34.66@KRBTEST.COM; defaulting to no policy
I20260619 14:51:45.646581 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:45.639433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:40695 (local address 127.19.34.126:33109)
0619 14:51:45.639543 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:45.639548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:45.639670 (+ 122us) server_negotiation.cc:407] Connection header received
0619 14:51:45.639741 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:45.639743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:45.639787 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:45.639880 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:45.640275 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:45.640780 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:45.641801 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:45.642284 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:45.642401 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:45.643483 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:45.643486 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:45.643489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:45.643519 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:45.645197 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:45.645620 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:45.645622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:45.645625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:45.645715 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:45.645982 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:45.645985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:45.645987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:45.646135 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:45.646207 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:45.646418 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 14:51:45.646494 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:51:45.646582 21547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:45.639327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:33109 (local address 127.19.34.65:40695)
0619 14:51:45.639585 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:45.639596 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:45.639676 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:45.639895 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:45.639898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:45.640075 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:45.640158 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:45.640163 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:45.640976 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:45.640979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:45.641666 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:45.641676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:45.642417 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:45.642421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:45.642515 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:45.643025 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:45.643037 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:45.643337 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:45.645326 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:45.645329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:45.645330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:45.645532 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:45.645798 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:45.645800 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:45.645801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:45.645872 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:45.646216 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:45.646218 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:45.646286 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 14:51:45.646367 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:51:45.646457 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":77,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
I20260619 14:51:45.661276 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:33109
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:45.759402 21549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:45.759603 21549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:45.759686 21549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:45.763571 21549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:45.763617 21549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:45.763697 21549 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 14:51:45.763737 21549 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:51:45.763804 21549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:51:45.768673 21549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.34.126:33109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:45.769477 21549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:45.770285 21549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:45.778967 21555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:45.778972 21554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:45.779222 21549 server_base.cc:1034] running on GCE node
W20260619 14:51:45.779335 21557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:45.779685 21549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:45.780236 21549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:45.781418 21549 hybrid_clock.cc:648] HybridClock initialized: now 1781880705781409 us; error 23 us; skew 500 ppm
Jun 19 14:51:45 dist-test-slave-3qdh krb5kdc[21292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880705, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:45.783865 21549 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:51:45.784667 21549 webserver.cc:460] Webserver started at http://127.19.34.66:41743/ using document root <none> and password file <none>
I20260619 14:51:45.785207 21549 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:45.785295 21549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:45.785501 21549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:45.787200 21549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "2f111686e0bf46fc801504db117cdb8a"
format_stamp: "Formatted at 2026-06-19 14:51:45 on dist-test-slave-3qdh"
server_key: "c308cda0644a8b2c152067f6a0b159f1"
server_key_iv: "412cd0031ccf09effda07d6a801b5cab"
server_key_version: "encryptionkey@0"
I20260619 14:51:45.787700 21549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "2f111686e0bf46fc801504db117cdb8a"
format_stamp: "Formatted at 2026-06-19 14:51:45 on dist-test-slave-3qdh"
server_key: "c308cda0644a8b2c152067f6a0b159f1"
server_key_iv: "412cd0031ccf09effda07d6a801b5cab"
server_key_version: "encryptionkey@0"
I20260619 14:51:45.791052 21549 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260619 14:51:45.793197 21564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:45.794301 21549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:45.794417 21549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "2f111686e0bf46fc801504db117cdb8a"
format_stamp: "Formatted at 2026-06-19 14:51:45 on dist-test-slave-3qdh"
server_key: "c308cda0644a8b2c152067f6a0b159f1"
server_key_iv: "412cd0031ccf09effda07d6a801b5cab"
server_key_version: "encryptionkey@0"
I20260619 14:51:45.794509 21549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:45.811851 21549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:45.814538 21549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:45.814698 21549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:45.815831 21549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:45.815871 21549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:45.815901 21549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:45.815953 21549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:51:45 dist-test-slave-3qdh krb5kdc[21292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880705, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:45.825789 21549 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:41937
I20260619 14:51:45.825879 21679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:41937 every 8 connection(s)
I20260619 14:51:45.826525 21549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:51:45.828882 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:45.816464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:33999 (local address 127.19.34.126:33109)
0619 14:51:45.816613 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:45.816619 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:45.817289 (+ 670us) server_negotiation.cc:407] Connection header received
0619 14:51:45.818289 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:45.818293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:45.818351 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:45.818411 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:45.819885 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:45.820619 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:45.822472 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:45.822782 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:45.822868 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:45.825729 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:45.825732 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:45.825733 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:45.825776 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:45.827209 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:45.827635 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:45.827637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:45.827638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:45.827709 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:45.827975 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:45.827977 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:45.827978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:45.828122 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:45.828189 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:45.828677 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 14:51:45.828773 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:51:45.829566 21573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:45.816763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:33109 (local address 127.19.34.66:33999)
0619 14:51:45.817143 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:45.817183 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:45.818054 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:45.818820 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:45.818830 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:45.819195 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:45.819732 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:45.819747 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:45.820741 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:45.820747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:45.822348 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:45.822360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:45.822925 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:45.822932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:45.823141 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:45.823782 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:45.823808 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:45.825549 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:45.827333 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:45.827336 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:45.827344 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:45.827553 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:45.827773 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:45.827775 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:45.827776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:45.827887 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:45.828205 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:45.828209 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:45.828457 (+ 248us) client_negotiation.cc:769] Sending connection context
0619 14:51:45.828623 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 14:51:45.828821 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":118,"threads_started":1}
I20260619 14:51:45.830706 21680 heartbeater.cc:347] Connected to a master server at 127.19.34.126:33109
I20260619 14:51:45.830830 21680 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:45.831300 21680 heartbeater.cc:511] Master 127.19.34.126:33109 requested a full tablet report, sending...
I20260619 14:51:45.832252 21336 ts_manager.cc:194] Registered new tserver with Master: 2f111686e0bf46fc801504db117cdb8a (127.19.34.66:41937)
I20260619 14:51:45.832969 21336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:33999
I20260619 14:51:45.835940 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 21549
I20260619 14:51:45.836063 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:51:45.836372 19593 external_mini_cluster.cc:1428] Setting key e922e78a4e60a1063f0a4ddc8a9b73db
I20260619 14:51:45.839038 21686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:45.832694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:33109 (local address 127.19.34.66:55871)
0619 14:51:45.832938 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:45.832952 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:45.833054 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:45.833318 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:45.833321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:45.833522 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:45.833634 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:45.833649 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:45.834361 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:45.834364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:45.834985 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:45.834994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:45.835382 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:45.835384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:45.835446 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:45.835868 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:45.835878 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:45.836231 (+ 353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:45.837708 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:45.837711 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:45.837712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:45.837878 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:45.838127 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:45.838128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:45.838129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:45.838187 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:45.838673 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:45.838676 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:45.838742 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 14:51:45.838823 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:51:45.838910 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":102,"threads_started":1}
I20260619 14:51:45.839253 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:45.832754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:55871 (local address 127.19.34.126:33109)
0619 14:51:45.832867 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:45.832871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:45.833021 (+ 150us) server_negotiation.cc:407] Connection header received
0619 14:51:45.833144 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:45.833146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:45.833191 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:45.833282 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:45.833754 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:45.834242 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:45.835082 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:45.835294 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:45.835365 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:45.836358 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:45.836361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:45.836364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:45.836388 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:45.837607 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:45.837965 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:45.837970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:45.837972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:45.838049 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:45.838262 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:45.838268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:45.838272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:45.838534 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:45.838670 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:45.838901 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 14:51:45.839022 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.19.34.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
I20260619 14:51:45.859050 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:33109
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:45.956686 21689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:45.956907 21689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:45.956948 21689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:45.960834 21689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:45.960878 21689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:45.960916 21689 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 14:51:45.960930 21689 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:51:45.960960 21689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:51:45.965747 21689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.34.126:33109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:45.966594 21689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:45.967525 21689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:45.976385 21694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:45.976431 21697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:45.976498 21689 server_base.cc:1034] running on GCE node
W20260619 14:51:45.976385 21695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:45.977146 21689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:45.977717 21689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:45.978904 21689 hybrid_clock.cc:648] HybridClock initialized: now 1781880705978887 us; error 30 us; skew 500 ppm
Jun 19 14:51:45 dist-test-slave-3qdh krb5kdc[21292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880705, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:45.981591 21689 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:51:45.982388 21689 webserver.cc:460] Webserver started at http://127.19.34.67:46009/ using document root <none> and password file <none>
I20260619 14:51:45.982892 21689 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:45.982977 21689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:45.983191 21689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:45.985049 21689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "e8f3c679f66046fdb39416b316bd6ab4"
format_stamp: "Formatted at 2026-06-19 14:51:45 on dist-test-slave-3qdh"
server_key: "4cdfb20686fc1387237c3b95685b7d91"
server_key_iv: "998fc6e6dba93fa7f8f6d59ca5058be7"
server_key_version: "encryptionkey@0"
I20260619 14:51:45.985550 21689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "e8f3c679f66046fdb39416b316bd6ab4"
format_stamp: "Formatted at 2026-06-19 14:51:45 on dist-test-slave-3qdh"
server_key: "4cdfb20686fc1387237c3b95685b7d91"
server_key_iv: "998fc6e6dba93fa7f8f6d59ca5058be7"
server_key_version: "encryptionkey@0"
I20260619 14:51:45.988886 21689 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:51:45.990899 21704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:45.991885 21689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:45.992009 21689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "e8f3c679f66046fdb39416b316bd6ab4"
format_stamp: "Formatted at 2026-06-19 14:51:45 on dist-test-slave-3qdh"
server_key: "4cdfb20686fc1387237c3b95685b7d91"
server_key_iv: "998fc6e6dba93fa7f8f6d59ca5058be7"
server_key_version: "encryptionkey@0"
I20260619 14:51:45.992110 21689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:46.013995 21689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:46.016788 21689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:46.016964 21689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:46.018266 21689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:46.018333 21689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:46.018392 21689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:46.018441 21689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:51:46 dist-test-slave-3qdh krb5kdc[21292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880705, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:46.028482 21689 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:46643
I20260619 14:51:46.028560 21819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:46643 every 8 connection(s)
I20260619 14:51:46.029496 21689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:51:46.031351 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.019005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:37207 (local address 127.19.34.126:33109)
0619 14:51:46.019166 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:46.019171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:46.019965 (+ 794us) server_negotiation.cc:407] Connection header received
0619 14:51:46.020871 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:46.020874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:46.020928 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:46.021002 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:46.022413 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.023223 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.024951 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.025169 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.025282 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.027652 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:46.027656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:46.027658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:46.027682 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:46.029471 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:46.029975 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:46.029978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:46.029980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:46.030060 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:46.030432 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:46.030434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:46.030435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:46.030597 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:46.030670 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:46.031153 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 14:51:46.031247 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:51:46.032087 21713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.019384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:33109 (local address 127.19.34.67:37207)
0619 14:51:46.019807 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:46.019850 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:46.020686 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:46.021290 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:46.021311 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:46.021706 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:46.022248 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:46.022266 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.023360 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.023364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:46.024831 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:46.024844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.025347 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.025351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:46.025515 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.026028 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:46.026042 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:46.027505 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:46.029594 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:46.029599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:46.029612 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:46.029841 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:46.030187 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:46.030191 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:46.030193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:46.030340 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:46.030680 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:46.030684 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:46.030919 (+ 235us) client_negotiation.cc:769] Sending connection context
0619 14:51:46.031109 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 14:51:46.031323 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":120,"threads_started":1}
I20260619 14:51:46.033212 21820 heartbeater.cc:347] Connected to a master server at 127.19.34.126:33109
I20260619 14:51:46.033353 21820 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:46.033774 21820 heartbeater.cc:511] Master 127.19.34.126:33109 requested a full tablet report, sending...
I20260619 14:51:46.034008 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 21689
I20260619 14:51:46.034109 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:51:46.034390 19593 external_mini_cluster.cc:1428] Setting key 66f5982cacd639ad095611bf427157bb
I20260619 14:51:46.034770 21337 ts_manager.cc:194] Registered new tserver with Master: e8f3c679f66046fdb39416b316bd6ab4 (127.19.34.67:46643)
I20260619 14:51:46.035319 21337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:37207
I20260619 14:51:46.036777 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:51:46.041522 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.035521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:44489 (local address 127.19.34.126:33109)
0619 14:51:46.035604 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:46.035607 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:46.035617 (+ 10us) server_negotiation.cc:407] Connection header received
0619 14:51:46.035643 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:46.035645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:46.035677 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:46.035743 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:46.036164 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.036751 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.037483 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.037706 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.037823 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.038817 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:46.038820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:46.038822 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:46.038844 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:46.040094 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:46.040525 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:46.040528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:46.040529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:46.040600 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:46.040836 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:46.040840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:46.040842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:46.040982 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:46.041072 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:46.041315 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 14:51:46.041405 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260619 14:51:46.041533 21826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.035237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:33109 (local address 127.19.34.67:44489)
0619 14:51:46.035438 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:46.035450 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:46.035514 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:46.035764 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:46.035783 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:46.035959 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:46.036064 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:46.036071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.036862 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.036867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:46.037378 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:46.037385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.037846 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.037849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:46.037933 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.038405 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:46.038424 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:46.038684 (+ 260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:46.040199 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:46.040202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:46.040203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:46.040436 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:46.040683 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:46.040686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:46.040687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:46.040761 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:46.041066 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:46.041069 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:46.041141 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 14:51:46.041266 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 14:51:46.041383 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
Jun 19 14:51:46 dist-test-slave-3qdh krb5kdc[21292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:51:46 dist-test-slave-3qdh krb5kdc[21292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:46.060196 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.053073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49376 (local address 127.19.34.126:33109)
0619 14:51:46.053199 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:46.053201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:46.053217 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:51:46.053243 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:46.053245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:46.053273 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:46.053357 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:46.053984 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.054436 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.055164 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.055356 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.055459 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.057423 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:46.057425 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:46.057427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:46.057445 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:46.058709 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:46.059106 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:46.059110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:46.059112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:46.059185 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:46.059459 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:46.059463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:46.059464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:46.059616 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:46.059747 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:46.059968 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 14:51:46.060062 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:51:46.062492 21337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:51:46.065260 21337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:51:46.079828 21837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.073546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:46643 (local address 127.0.0.1:49014)
0619 14:51:46.074018 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:46.074064 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:46.074198 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:46.075183 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:46.075185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:46.075201 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:46.075300 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:46.075305 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.076705 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.076710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:46.078284 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:46.078294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.079323 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.079327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:46.079437 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.079459 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:51:46.079552 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 14:51:46.079657 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":130,"thread_start_us":106,"threads_started":1}
I20260619 14:51:46.079852 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.074197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:33029 (local address 127.0.0.1:58172)
0619 14:51:46.074614 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:46.074639 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:46.074726 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:46.075883 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:46.075886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:46.075905 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:46.076014 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:46.076020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.077361 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.077366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:46.078150 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:46.078160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.079281 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.079285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:46.079411 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.079457 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 14:51:46.079609 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 14:51:46.079690 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":108,"threads_started":1}
I20260619 14:51:46.079876 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.074952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58172 (local address 127.19.34.65:33029)
0619 14:51:46.075386 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:46.075399 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:46.075441 (+ 42us) server_negotiation.cc:407] Connection header received
0619 14:51:46.075516 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:46.075521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:46.075717 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:46.076165 (+ 448us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:46.076319 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.077192 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.078343 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.079144 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.079282 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.079442 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:46.079645 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 14:51:46.079740 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":115,"threads_started":1}
I20260619 14:51:46.080132 21839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.074003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:41937 (local address 127.0.0.1:39420)
0619 14:51:46.074441 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:46.074455 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:46.074534 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:46.075019 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:46.075024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:46.075044 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:46.075173 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:46.075179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.076757 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.076760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:46.078284 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:46.078294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.079323 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.079327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:46.079823 (+ 496us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.079853 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 14:51:46.079929 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:51:46.079995 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":515,"thread_start_us":94,"threads_started":1}
I20260619 14:51:46.080550 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.074201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39420 (local address 127.19.34.66:41937)
0619 14:51:46.074528 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:46.074542 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:46.074569 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:51:46.074648 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:46.074656 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:46.074863 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:46.075020 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:46.075294 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.076613 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.078424 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.079179 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.079372 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.079531 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:46.080283 (+ 752us) server_negotiation.cc:299] Negotiation successful
0619 14:51:46.080407 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":146,"threads_started":1}
I20260619 14:51:46.080724 21841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.074508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49014 (local address 127.19.34.67:46643)
0619 14:51:46.074767 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:46.074779 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:46.074797 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:51:46.074860 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:46.074865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:46.075043 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:46.075440 (+ 397us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:46.075590 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.076583 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.078424 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.079179 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.080329 (+ 1150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.080462 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:46.080507 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 14:51:46.080589 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":100,"threads_started":1}
I20260619 14:51:46.083020 21614 tablet_service.cc:1467] Processing CreateTablet for tablet 484ef9af77184e738eaf98d1d38dc5a8 (DEFAULT_TABLE table=test-table [id=bb2f248ba9cc4e0883b4ca1fbdc9f57e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:46.083153 21474 tablet_service.cc:1467] Processing CreateTablet for tablet 484ef9af77184e738eaf98d1d38dc5a8 (DEFAULT_TABLE table=test-table [id=bb2f248ba9cc4e0883b4ca1fbdc9f57e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:46.083153 21754 tablet_service.cc:1467] Processing CreateTablet for tablet 484ef9af77184e738eaf98d1d38dc5a8 (DEFAULT_TABLE table=test-table [id=bb2f248ba9cc4e0883b4ca1fbdc9f57e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:46.083945 21614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 484ef9af77184e738eaf98d1d38dc5a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:46.084128 21474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 484ef9af77184e738eaf98d1d38dc5a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:46.084146 21754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 484ef9af77184e738eaf98d1d38dc5a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:46.090626 21843 tablet_bootstrap.cc:492] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a: Bootstrap starting.
I20260619 14:51:46.091531 21844 tablet_bootstrap.cc:492] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4: Bootstrap starting.
I20260619 14:51:46.092540 21845 tablet_bootstrap.cc:492] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad: Bootstrap starting.
I20260619 14:51:46.093143 21843 tablet_bootstrap.cc:654] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:46.093551 21844 tablet_bootstrap.cc:654] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:46.093971 21843 log.cc:824] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:46.094322 21844 log.cc:824] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:46.094372 21845 tablet_bootstrap.cc:654] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:46.095199 21845 log.cc:824] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:46.096010 21843 tablet_bootstrap.cc:492] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a: No bootstrap required, opened a new log
I20260619 14:51:46.096051 21844 tablet_bootstrap.cc:492] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4: No bootstrap required, opened a new log
I20260619 14:51:46.096284 21844 ts_tablet_manager.cc:1397] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260619 14:51:46.096315 21843 ts_tablet_manager.cc:1397] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260619 14:51:46.097026 21845 tablet_bootstrap.cc:492] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad: No bootstrap required, opened a new log
I20260619 14:51:46.097204 21845 ts_tablet_manager.cc:1397] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:51:46.100004 21844 raft_consensus.cc:348] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } } peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } }
I20260619 14:51:46.100004 21843 raft_consensus.cc:348] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } } peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } }
I20260619 14:51:46.100230 21843 raft_consensus.cc:374] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:46.100224 21844 raft_consensus.cc:374] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:46.100278 21844 raft_consensus.cc:729] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f3c679f66046fdb39416b316bd6ab4, State: Initialized, Role: FOLLOWER
I20260619 14:51:46.100278 21843 raft_consensus.cc:729] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f111686e0bf46fc801504db117cdb8a, State: Initialized, Role: FOLLOWER
I20260619 14:51:46.100471 21845 raft_consensus.cc:348] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } } peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } }
I20260619 14:51:46.100721 21845 raft_consensus.cc:374] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:46.100672 21843 consensus_queue.cc:260] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } } peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } }
I20260619 14:51:46.100672 21844 consensus_queue.cc:260] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } } peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } }
I20260619 14:51:46.100782 21845 raft_consensus.cc:729] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d56944bba9847c4b319a72f051808ad, State: Initialized, Role: FOLLOWER
I20260619 14:51:46.101171 21845 consensus_queue.cc:260] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } } peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } }
I20260619 14:51:46.101506 21844 ts_tablet_manager.cc:1428] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260619 14:51:46.101506 21843 ts_tablet_manager.cc:1428] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260619 14:51:46.101797 21820 heartbeater.cc:503] Master 127.19.34.126:33109 was elected leader, sending a full tablet report...
I20260619 14:51:46.101797 21680 heartbeater.cc:503] Master 127.19.34.126:33109 was elected leader, sending a full tablet report...
I20260619 14:51:46.101814 21540 heartbeater.cc:503] Master 127.19.34.126:33109 was elected leader, sending a full tablet report...
I20260619 14:51:46.102063 21845 ts_tablet_manager.cc:1428] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260619 14:51:46.134949 21541 tablet.cc:2369] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:51:46.280401 21821 tablet.cc:2369] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:51:46.327595 21681 tablet.cc:2369] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:51:46.452672 21849 raft_consensus.cc:482] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:51:46.452847 21849 raft_consensus.cc:504] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } } peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } }
I20260619 14:51:46.453922 21849 leader_election.cc:288] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d56944bba9847c4b319a72f051808ad (127.19.34.65:33029), 2f111686e0bf46fc801504db117cdb8a (127.19.34.66:41937)
I20260619 14:51:46.457412 21842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.454278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:53213 (local address 127.19.34.65:33029)
0619 14:51:46.454415 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:46.454419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:46.454453 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:51:46.454507 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:46.454510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:46.454555 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:46.454630 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:46.454927 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.455492 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.456582 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.456935 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.457039 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.457087 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:46.457200 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 14:51:46.457277 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:51:46.457556 21713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.454127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:33029 (local address 127.19.34.67:53213)
0619 14:51:46.454275 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:46.454287 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:46.454386 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:46.454668 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:46.454671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:46.454694 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:46.454813 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:46.454820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.455593 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.455595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:46.456471 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:46.456481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.457031 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.457033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:46.457097 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.457114 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 14:51:46.457171 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 14:51:46.457221 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":1}
I20260619 14:51:46.458103 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.454277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:51623 (local address 127.19.34.66:41937)
0619 14:51:46.454406 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:46.454412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:46.454751 (+ 339us) server_negotiation.cc:407] Connection header received
0619 14:51:46.454837 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:46.454839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:46.454899 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:46.454969 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:46.455530 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.455969 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.456964 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.457481 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.457598 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.457644 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:46.457948 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 14:51:46.458008 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:51:46.458127 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.454152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:41937 (local address 127.19.34.67:51623)
0619 14:51:46.454654 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:46.454673 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:46.454752 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:46.454991 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:46.454993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:46.455006 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:46.455392 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:46.455396 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.456066 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.456069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:46.456860 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:46.456867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.457639 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.457642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:46.457829 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.457865 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 14:51:46.457931 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 14:51:46.457995 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":176,"thread_start_us":116,"threads_started":1}
I20260619 14:51:46.458344 21494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "484ef9af77184e738eaf98d1d38dc5a8" candidate_uuid: "e8f3c679f66046fdb39416b316bd6ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d56944bba9847c4b319a72f051808ad" is_pre_election: true
I20260619 14:51:46.458616 21494 raft_consensus.cc:2454] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f3c679f66046fdb39416b316bd6ab4 in term 0.
I20260619 14:51:46.458660 21634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "484ef9af77184e738eaf98d1d38dc5a8" candidate_uuid: "e8f3c679f66046fdb39416b316bd6ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f111686e0bf46fc801504db117cdb8a" is_pre_election: true
I20260619 14:51:46.458889 21634 raft_consensus.cc:2454] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f3c679f66046fdb39416b316bd6ab4 in term 0.
I20260619 14:51:46.459126 21708 leader_election.cc:302] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d56944bba9847c4b319a72f051808ad, e8f3c679f66046fdb39416b316bd6ab4; no voters:
I20260619 14:51:46.459388 21849 raft_consensus.cc:2781] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:51:46.459497 21849 raft_consensus.cc:482] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:51:46.459561 21849 raft_consensus.cc:3037] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:46.460700 21849 raft_consensus.cc:504] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } } peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } }
I20260619 14:51:46.461005 21849 leader_election.cc:288] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 4d56944bba9847c4b319a72f051808ad (127.19.34.65:33029), 2f111686e0bf46fc801504db117cdb8a (127.19.34.66:41937)
I20260619 14:51:46.461308 21634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "484ef9af77184e738eaf98d1d38dc5a8" candidate_uuid: "e8f3c679f66046fdb39416b316bd6ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f111686e0bf46fc801504db117cdb8a"
I20260619 14:51:46.461319 21494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "484ef9af77184e738eaf98d1d38dc5a8" candidate_uuid: "e8f3c679f66046fdb39416b316bd6ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d56944bba9847c4b319a72f051808ad"
I20260619 14:51:46.461436 21634 raft_consensus.cc:3037] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:46.461422 21494 raft_consensus.cc:3037] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:46.462404 21494 raft_consensus.cc:2454] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f3c679f66046fdb39416b316bd6ab4 in term 1.
I20260619 14:51:46.462416 21634 raft_consensus.cc:2454] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f3c679f66046fdb39416b316bd6ab4 in term 1.
I20260619 14:51:46.462678 21708 leader_election.cc:302] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d56944bba9847c4b319a72f051808ad, e8f3c679f66046fdb39416b316bd6ab4; no voters:
I20260619 14:51:46.462826 21849 raft_consensus.cc:2781] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:46.463006 21849 raft_consensus.cc:686] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 [term 1 LEADER]: Becoming Leader. State: Replica: e8f3c679f66046fdb39416b316bd6ab4, State: Running, Role: LEADER
I20260619 14:51:46.463297 21849 consensus_queue.cc:237] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } } peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } }
I20260619 14:51:46.466353 21337 catalog_manager.cc:5495] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8f3c679f66046fdb39416b316bd6ab4 (127.19.34.67). New cstate: current_term: 1 leader_uuid: "e8f3c679f66046fdb39416b316bd6ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } health_report { overall_health: UNKNOWN } } }
Jun 19 14:51:46 dist-test-slave-3qdh krb5kdc[21292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880705, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:46.544673 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.537563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:33109 (local address 127.0.0.1:49388)
0619 14:51:46.537787 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:46.537797 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:46.537872 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:46.538088 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:46.538092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:46.538422 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:46.538503 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:46.538508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.539144 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.539146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:46.539604 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:46.539610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.539934 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.539936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:46.539990 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.540561 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:46.540571 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:46.542051 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:46.543415 (+ 1364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:46.543422 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:46.543426 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:46.543633 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:46.543856 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:46.543858 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:46.543860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:46.543930 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:46.544262 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:46.544266 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:46.544399 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 14:51:46.544478 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 14:51:46.544564 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":65,"threads_started":1}
I20260619 14:51:46.544819 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.537639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49388 (local address 127.19.34.126:33109)
0619 14:51:46.537768 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:46.537772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:46.537861 (+ 89us) server_negotiation.cc:407] Connection header received
0619 14:51:46.537949 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:46.537951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:46.537984 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:46.538041 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:46.538590 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.539067 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.539687 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.539854 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.539918 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.542158 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:46.542161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:46.542163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:46.542185 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:46.543324 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:46.543713 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:46.543716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:46.543717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:46.543782 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:46.544001 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:46.544003 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:46.544005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:46.544144 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:46.544234 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:46.544546 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 14:51:46.544647 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:51:46.547904 21337 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:49388:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 14:51:46.548571 21337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:51:46.552840 21474 tablet_service.cc:1467] Processing CreateTablet for tablet d02268d3cdd741c38d5de76fef47784f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b5d0000fd9454d4c852febc2112ad58f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:51:46.552991 21754 tablet_service.cc:1467] Processing CreateTablet for tablet d02268d3cdd741c38d5de76fef47784f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b5d0000fd9454d4c852febc2112ad58f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:51:46.552855 21614 tablet_service.cc:1467] Processing CreateTablet for tablet d02268d3cdd741c38d5de76fef47784f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b5d0000fd9454d4c852febc2112ad58f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:51:46.553514 21474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d02268d3cdd741c38d5de76fef47784f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:46.553522 21754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d02268d3cdd741c38d5de76fef47784f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:46.553524 21614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d02268d3cdd741c38d5de76fef47784f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:46.555729 21845 tablet_bootstrap.cc:492] T d02268d3cdd741c38d5de76fef47784f P 4d56944bba9847c4b319a72f051808ad: Bootstrap starting.
I20260619 14:51:46.556370 21844 tablet_bootstrap.cc:492] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4: Bootstrap starting.
I20260619 14:51:46.556370 21843 tablet_bootstrap.cc:492] T d02268d3cdd741c38d5de76fef47784f P 2f111686e0bf46fc801504db117cdb8a: Bootstrap starting.
I20260619 14:51:46.556627 21845 tablet_bootstrap.cc:654] T d02268d3cdd741c38d5de76fef47784f P 4d56944bba9847c4b319a72f051808ad: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:46.557196 21843 tablet_bootstrap.cc:654] T d02268d3cdd741c38d5de76fef47784f P 2f111686e0bf46fc801504db117cdb8a: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:46.557219 21844 tablet_bootstrap.cc:654] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:46.557590 21845 tablet_bootstrap.cc:492] T d02268d3cdd741c38d5de76fef47784f P 4d56944bba9847c4b319a72f051808ad: No bootstrap required, opened a new log
I20260619 14:51:46.557686 21845 ts_tablet_manager.cc:1397] T d02268d3cdd741c38d5de76fef47784f P 4d56944bba9847c4b319a72f051808ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:51:46.558159 21845 raft_consensus.cc:348] T d02268d3cdd741c38d5de76fef47784f P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:46.558259 21845 raft_consensus.cc:374] T d02268d3cdd741c38d5de76fef47784f P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:46.558290 21845 raft_consensus.cc:729] T d02268d3cdd741c38d5de76fef47784f P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d56944bba9847c4b319a72f051808ad, State: Initialized, Role: FOLLOWER
I20260619 14:51:46.558394 21843 tablet_bootstrap.cc:492] T d02268d3cdd741c38d5de76fef47784f P 2f111686e0bf46fc801504db117cdb8a: No bootstrap required, opened a new log
I20260619 14:51:46.558393 21845 consensus_queue.cc:260] T d02268d3cdd741c38d5de76fef47784f P 4d56944bba9847c4b319a72f051808ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:46.558490 21843 ts_tablet_manager.cc:1397] T d02268d3cdd741c38d5de76fef47784f P 2f111686e0bf46fc801504db117cdb8a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260619 14:51:46.558640 21845 ts_tablet_manager.cc:1428] T d02268d3cdd741c38d5de76fef47784f P 4d56944bba9847c4b319a72f051808ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:51:46.558641 21844 tablet_bootstrap.cc:492] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4: No bootstrap required, opened a new log
I20260619 14:51:46.558704 21844 ts_tablet_manager.cc:1397] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:51:46.558866 21851 tablet_replica.cc:439] T d02268d3cdd741c38d5de76fef47784f P 4d56944bba9847c4b319a72f051808ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } } }
I20260619 14:51:46.558903 21843 raft_consensus.cc:348] T d02268d3cdd741c38d5de76fef47784f P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:46.558974 21851 tablet_replica.cc:442] T d02268d3cdd741c38d5de76fef47784f P 4d56944bba9847c4b319a72f051808ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:51:46.558992 21843 raft_consensus.cc:374] T d02268d3cdd741c38d5de76fef47784f P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:46.559022 21843 raft_consensus.cc:729] T d02268d3cdd741c38d5de76fef47784f P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f111686e0bf46fc801504db117cdb8a, State: Initialized, Role: FOLLOWER
I20260619 14:51:46.559068 21844 raft_consensus.cc:348] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:46.559128 21843 consensus_queue.cc:260] T d02268d3cdd741c38d5de76fef47784f P 2f111686e0bf46fc801504db117cdb8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:46.559185 21844 raft_consensus.cc:374] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:46.559216 21844 raft_consensus.cc:729] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f3c679f66046fdb39416b316bd6ab4, State: Initialized, Role: FOLLOWER
I20260619 14:51:46.559315 21844 consensus_queue.cc:260] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:46.559355 21850 tablet_replica.cc:439] T d02268d3cdd741c38d5de76fef47784f P 2f111686e0bf46fc801504db117cdb8a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } } }
I20260619 14:51:46.559497 21850 tablet_replica.cc:442] T d02268d3cdd741c38d5de76fef47784f P 2f111686e0bf46fc801504db117cdb8a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:51:46.559538 21844 ts_tablet_manager.cc:1428] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:51:46.559931 21849 tablet_replica.cc:439] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } } }
I20260619 14:51:46.560038 21849 tablet_replica.cc:442] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:51:46.559360 21843 ts_tablet_manager.cc:1428] T d02268d3cdd741c38d5de76fef47784f P 2f111686e0bf46fc801504db117cdb8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:51:46.588676 21849 raft_consensus.cc:482] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:51:46.588757 21849 raft_consensus.cc:504] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:46.589082 21849 leader_election.cc:288] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d56944bba9847c4b319a72f051808ad (127.19.34.65:33029), 2f111686e0bf46fc801504db117cdb8a (127.19.34.66:41937)
I20260619 14:51:46.589376 21494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02268d3cdd741c38d5de76fef47784f" candidate_uuid: "e8f3c679f66046fdb39416b316bd6ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d56944bba9847c4b319a72f051808ad" is_pre_election: true
I20260619 14:51:46.589519 21494 raft_consensus.cc:2454] T d02268d3cdd741c38d5de76fef47784f P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f3c679f66046fdb39416b316bd6ab4 in term 0.
I20260619 14:51:46.589480 21634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02268d3cdd741c38d5de76fef47784f" candidate_uuid: "e8f3c679f66046fdb39416b316bd6ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f111686e0bf46fc801504db117cdb8a" is_pre_election: true
I20260619 14:51:46.589628 21634 raft_consensus.cc:2454] T d02268d3cdd741c38d5de76fef47784f P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f3c679f66046fdb39416b316bd6ab4 in term 0.
I20260619 14:51:46.589810 21708 leader_election.cc:302] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d56944bba9847c4b319a72f051808ad, e8f3c679f66046fdb39416b316bd6ab4; no voters:
I20260619 14:51:46.589977 21849 raft_consensus.cc:2781] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:51:46.590025 21849 raft_consensus.cc:482] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:51:46.590049 21849 raft_consensus.cc:3037] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:46.590782 21849 raft_consensus.cc:504] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:46.591034 21849 leader_election.cc:288] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 4d56944bba9847c4b319a72f051808ad (127.19.34.65:33029), 2f111686e0bf46fc801504db117cdb8a (127.19.34.66:41937)
I20260619 14:51:46.591342 21634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02268d3cdd741c38d5de76fef47784f" candidate_uuid: "e8f3c679f66046fdb39416b316bd6ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f111686e0bf46fc801504db117cdb8a"
I20260619 14:51:46.591323 21494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d02268d3cdd741c38d5de76fef47784f" candidate_uuid: "e8f3c679f66046fdb39416b316bd6ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d56944bba9847c4b319a72f051808ad"
I20260619 14:51:46.591440 21494 raft_consensus.cc:3037] T d02268d3cdd741c38d5de76fef47784f P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:46.591449 21634 raft_consensus.cc:3037] T d02268d3cdd741c38d5de76fef47784f P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:46.592197 21634 raft_consensus.cc:2454] T d02268d3cdd741c38d5de76fef47784f P 2f111686e0bf46fc801504db117cdb8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f3c679f66046fdb39416b316bd6ab4 in term 1.
I20260619 14:51:46.592406 21494 raft_consensus.cc:2454] T d02268d3cdd741c38d5de76fef47784f P 4d56944bba9847c4b319a72f051808ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f3c679f66046fdb39416b316bd6ab4 in term 1.
I20260619 14:51:46.592456 21705 leader_election.cc:302] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f111686e0bf46fc801504db117cdb8a, e8f3c679f66046fdb39416b316bd6ab4; no voters:
I20260619 14:51:46.592602 21849 raft_consensus.cc:2781] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:46.592685 21849 raft_consensus.cc:686] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4 [term 1 LEADER]: Becoming Leader. State: Replica: e8f3c679f66046fdb39416b316bd6ab4, State: Running, Role: LEADER
I20260619 14:51:46.592794 21849 consensus_queue.cc:237] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:46.593449 21853 tablet_replica.cc:439] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4: TxnStatusTablet state changed. Reason: New leader e8f3c679f66046fdb39416b316bd6ab4. Latest consensus state: current_term: 1 leader_uuid: "e8f3c679f66046fdb39416b316bd6ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } } }
I20260619 14:51:46.593577 21853 tablet_replica.cc:442] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:51:46.593819 21865 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 14:51:46.594030 21865 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 14:51:46.594313 21335 catalog_manager.cc:5495] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8f3c679f66046fdb39416b316bd6ab4 (127.19.34.67). New cstate: current_term: 1 leader_uuid: "e8f3c679f66046fdb39416b316bd6ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } health_report { overall_health: HEALTHY } } }
I20260619 14:51:46.612387 21860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.608768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:46643 (local address 127.0.0.1:42842)
0619 14:51:46.608907 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:46.608921 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:46.609011 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:46.609293 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:46.609295 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:46.609303 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:51:46.609422 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:46.609427 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.610073 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.610076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:46.611005 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:46.611014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.611426 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.611430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:46.611510 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.611555 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:51:46.612087 (+ 532us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:51:46.612099 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 14:51:46.612163 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 14:51:46.612241 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 14:51:46.612403 21868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.608869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42842 (local address 127.19.34.67:46643)
0619 14:51:46.609060 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:46.609066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:46.609077 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:51:46.609123 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:46.609124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:46.609178 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:46.609256 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:46.609515 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.609984 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.611104 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.611328 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.611395 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.611683 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:46.611995 (+ 312us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:46.612067 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:46.612195 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 14:51:46.612233 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260619 14:51:46.616446 21494 raft_consensus.cc:1261] T d02268d3cdd741c38d5de76fef47784f P 4d56944bba9847c4b319a72f051808ad [term 1 FOLLOWER]: Refusing update from remote peer e8f3c679f66046fdb39416b316bd6ab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:46.616447 21634 raft_consensus.cc:1261] T d02268d3cdd741c38d5de76fef47784f P 2f111686e0bf46fc801504db117cdb8a [term 1 FOLLOWER]: Refusing update from remote peer e8f3c679f66046fdb39416b316bd6ab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:46.617105 21853 consensus_queue.cc:1035] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:51:46.617309 21849 consensus_queue.cc:1035] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:51:46.621273 21851 tablet_replica.cc:439] T d02268d3cdd741c38d5de76fef47784f P 4d56944bba9847c4b319a72f051808ad: TxnStatusTablet state changed. Reason: New leader e8f3c679f66046fdb39416b316bd6ab4. Latest consensus state: current_term: 1 leader_uuid: "e8f3c679f66046fdb39416b316bd6ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } } }
I20260619 14:51:46.621393 21851 tablet_replica.cc:442] T d02268d3cdd741c38d5de76fef47784f P 4d56944bba9847c4b319a72f051808ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:51:46.622143 21850 tablet_replica.cc:439] T d02268d3cdd741c38d5de76fef47784f P 2f111686e0bf46fc801504db117cdb8a: TxnStatusTablet state changed. Reason: New leader e8f3c679f66046fdb39416b316bd6ab4. Latest consensus state: current_term: 1 leader_uuid: "e8f3c679f66046fdb39416b316bd6ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } } }
I20260619 14:51:46.622251 21850 tablet_replica.cc:442] T d02268d3cdd741c38d5de76fef47784f P 2f111686e0bf46fc801504db117cdb8a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:51:46.625185 21849 tablet_replica.cc:439] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8f3c679f66046fdb39416b316bd6ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } } }
I20260619 14:51:46.625272 21849 tablet_replica.cc:442] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:51:46.625555 21851 tablet_replica.cc:439] T d02268d3cdd741c38d5de76fef47784f P 4d56944bba9847c4b319a72f051808ad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8f3c679f66046fdb39416b316bd6ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } } }
I20260619 14:51:46.625656 21851 tablet_replica.cc:442] T d02268d3cdd741c38d5de76fef47784f P 4d56944bba9847c4b319a72f051808ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:51:46.627830 21870 mvcc.cc:205] Tried to move back new op lower bound from 7298583374295244800 to 7298583374205480960. Current Snapshot: MvccSnapshot[applied={T|T < 7298583374295244800 or (T in {7298583374295244800})}]
I20260619 14:51:46.629690 21853 tablet_replica.cc:439] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8f3c679f66046fdb39416b316bd6ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } } }
I20260619 14:51:46.629794 21853 tablet_replica.cc:442] T d02268d3cdd741c38d5de76fef47784f P e8f3c679f66046fdb39416b316bd6ab4: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:51:46.630424 21850 tablet_replica.cc:439] T d02268d3cdd741c38d5de76fef47784f P 2f111686e0bf46fc801504db117cdb8a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8f3c679f66046fdb39416b316bd6ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } } }
I20260619 14:51:46.630537 21850 tablet_replica.cc:442] T d02268d3cdd741c38d5de76fef47784f P 2f111686e0bf46fc801504db117cdb8a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:51:46.630441 21871 mvcc.cc:205] Tried to move back new op lower bound from 7298583374295244800 to 7298583374205480960. Current Snapshot: MvccSnapshot[applied={T|T < 7298583374295244800}]
I20260619 14:51:46.635143 21868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.630177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42854 (local address 127.19.34.67:46643)
0619 14:51:46.630367 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:46.630371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:46.631617 (+ 1246us) server_negotiation.cc:407] Connection header received
0619 14:51:46.631776 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:46.631782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:46.631821 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:46.631901 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:46.632130 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.632816 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.634114 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.634459 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.634572 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.634802 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:46.634865 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:46.634927 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:46.635007 (+ 80us) server_negotiation.cc:299] Negotiation successful
0619 14:51:46.635046 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260619 14:51:46.650112 21868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.647200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:35895 (local address 127.19.34.67:46643)
0619 14:51:46.647296 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:46.647300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:46.647440 (+ 140us) server_negotiation.cc:407] Connection header received
0619 14:51:46.647547 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:46.647552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:46.647584 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:46.647656 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:46.647919 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.648414 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.649146 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.649345 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.649440 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.649736 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:46.649804 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:46.649875 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:46.649947 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 14:51:46.649989 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":35}
I20260619 14:51:46.650125 21881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:46.647126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:46643 (local address 127.19.34.67:35895)
0619 14:51:46.647367 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:46.647384 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:46.647480 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:46.647662 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:46.647664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:46.647670 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:51:46.647807 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:46.647816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.648501 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.648506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:46.649046 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:46.649052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:46.649439 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:46.649441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:46.649558 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:46.649597 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:51:46.649877 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:51:46.649884 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:51:46.649940 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 14:51:46.650004 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":39,"thread_start_us":90,"threads_started":1}
I20260619 14:51:46.656076 21634 raft_consensus.cc:1261] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a [term 1 FOLLOWER]: Refusing update from remote peer e8f3c679f66046fdb39416b316bd6ab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:46.656072 21494 raft_consensus.cc:1261] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad [term 1 FOLLOWER]: Refusing update from remote peer e8f3c679f66046fdb39416b316bd6ab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:46.656455 21853 consensus_queue.cc:1035] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:51:46.656760 21853 consensus_queue.cc:1035] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:51:46.660924 21869 mvcc.cc:205] Tried to move back new op lower bound from 7298583374460125184 to 7298583373675868160. Current Snapshot: MvccSnapshot[applied={T|T < 7298583374460125184}]
I20260619 14:51:46.693414 21335 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:51:46.693581 21335 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:51:46.695585 21335 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de: Sending DeleteTablet for 3 replicas of tablet 484ef9af77184e738eaf98d1d38dc5a8
I20260619 14:51:46.696131 21754 tablet_service.cc:1514] Processing DeleteTablet for tablet 484ef9af77184e738eaf98d1d38dc5a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:46 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:49014
I20260619 14:51:46.696228 21474 tablet_service.cc:1514] Processing DeleteTablet for tablet 484ef9af77184e738eaf98d1d38dc5a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:46 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:58172
I20260619 14:51:46.696226 21614 tablet_service.cc:1514] Processing DeleteTablet for tablet 484ef9af77184e738eaf98d1d38dc5a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:46 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:39420
I20260619 14:51:46.696729 21887 tablet_replica.cc:330] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad: stopping tablet replica
I20260619 14:51:46.696768 21889 tablet_replica.cc:330] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a: stopping tablet replica
I20260619 14:51:46.696987 21889 raft_consensus.cc:2229] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:51:46.696969 21887 raft_consensus.cc:2229] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:51:46.697165 21887 raft_consensus.cc:2258] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:46.697153 21889 raft_consensus.cc:2258] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:46.697436 21888 tablet_replica.cc:330] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4: stopping tablet replica
I20260619 14:51:46.697479 21335 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 14:51:46.697664 21888 raft_consensus.cc:2229] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 [term 1 LEADER]: Raft consensus shutting down.
W20260619 14:51:46.697801 21335 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:51:46.697942 21888 raft_consensus.cc:2258] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:46.698827 21887 ts_tablet_manager.cc:1905] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:46.699024 21888 ts_tablet_manager.cc:1905] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:46.701956 21887 ts_tablet_manager.cc:1918] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:51:46.702041 21887 log.cc:1196] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-0/wal/wals/484ef9af77184e738eaf98d1d38dc5a8
I20260619 14:51:46.702050 21474 tablet_service.cc:1467] Processing CreateTablet for tablet b99d49335502422a80048e3626f92fb7 (DEFAULT_TABLE table=test-table [id=cc2c548bb4e2423b8188edeba8012bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:46.702294 21474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99d49335502422a80048e3626f92fb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:46.702330 21888 ts_tablet_manager.cc:1918] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:51:46.702404 21888 log.cc:1196] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/484ef9af77184e738eaf98d1d38dc5a8
I20260619 14:51:46.702425 21887 ts_tablet_manager.cc:1939] T 484ef9af77184e738eaf98d1d38dc5a8 P 4d56944bba9847c4b319a72f051808ad: Deleting consensus metadata
I20260619 14:51:46.702678 21888 ts_tablet_manager.cc:1939] T 484ef9af77184e738eaf98d1d38dc5a8 P e8f3c679f66046fdb39416b316bd6ab4: Deleting consensus metadata
I20260619 14:51:46.703495 21324 catalog_manager.cc:4841] TS 4d56944bba9847c4b319a72f051808ad (127.19.34.65:33029): tablet 484ef9af77184e738eaf98d1d38dc5a8 (table test-table [id=bb2f248ba9cc4e0883b4ca1fbdc9f57e]) successfully deleted
I20260619 14:51:46.704186 21845 tablet_bootstrap.cc:492] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad: Bootstrap starting.
I20260619 14:51:46.704491 21614 tablet_service.cc:1467] Processing CreateTablet for tablet b99d49335502422a80048e3626f92fb7 (DEFAULT_TABLE table=test-table [id=cc2c548bb4e2423b8188edeba8012bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:46.704737 21614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99d49335502422a80048e3626f92fb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:46.705011 21845 tablet_bootstrap.cc:654] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:46.706071 21845 tablet_bootstrap.cc:492] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad: No bootstrap required, opened a new log
I20260619 14:51:46.706128 21845 ts_tablet_manager.cc:1397] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:51:46.706468 21845 raft_consensus.cc:348] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:46.706566 21845 raft_consensus.cc:374] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:46.706593 21845 raft_consensus.cc:729] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d56944bba9847c4b319a72f051808ad, State: Initialized, Role: FOLLOWER
I20260619 14:51:46.706698 21845 consensus_queue.cc:260] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:46.706923 21845 ts_tablet_manager.cc:1428] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:51:46.707099 21843 tablet_bootstrap.cc:492] T b99d49335502422a80048e3626f92fb7 P 2f111686e0bf46fc801504db117cdb8a: Bootstrap starting.
I20260619 14:51:46.707304 21324 catalog_manager.cc:4841] TS e8f3c679f66046fdb39416b316bd6ab4 (127.19.34.67:46643): tablet 484ef9af77184e738eaf98d1d38dc5a8 (table test-table [id=bb2f248ba9cc4e0883b4ca1fbdc9f57e]) successfully deleted
I20260619 14:51:46.707806 21754 tablet_service.cc:1467] Processing CreateTablet for tablet b99d49335502422a80048e3626f92fb7 (DEFAULT_TABLE table=test-table [id=cc2c548bb4e2423b8188edeba8012bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:46.707917 21843 tablet_bootstrap.cc:654] T b99d49335502422a80048e3626f92fb7 P 2f111686e0bf46fc801504db117cdb8a: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:46.708040 21754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b99d49335502422a80048e3626f92fb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:46.710052 21844 tablet_bootstrap.cc:492] T b99d49335502422a80048e3626f92fb7 P e8f3c679f66046fdb39416b316bd6ab4: Bootstrap starting.
I20260619 14:51:46.710685 21843 tablet_bootstrap.cc:492] T b99d49335502422a80048e3626f92fb7 P 2f111686e0bf46fc801504db117cdb8a: No bootstrap required, opened a new log
I20260619 14:51:46.710736 21843 ts_tablet_manager.cc:1397] T b99d49335502422a80048e3626f92fb7 P 2f111686e0bf46fc801504db117cdb8a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 14:51:46.710867 21844 tablet_bootstrap.cc:654] T b99d49335502422a80048e3626f92fb7 P e8f3c679f66046fdb39416b316bd6ab4: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:46.710994 21843 raft_consensus.cc:348] T b99d49335502422a80048e3626f92fb7 P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:46.711056 21843 raft_consensus.cc:374] T b99d49335502422a80048e3626f92fb7 P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:46.711073 21843 raft_consensus.cc:729] T b99d49335502422a80048e3626f92fb7 P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f111686e0bf46fc801504db117cdb8a, State: Initialized, Role: FOLLOWER
I20260619 14:51:46.711180 21843 consensus_queue.cc:260] T b99d49335502422a80048e3626f92fb7 P 2f111686e0bf46fc801504db117cdb8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:46.711413 21843 ts_tablet_manager.cc:1428] T b99d49335502422a80048e3626f92fb7 P 2f111686e0bf46fc801504db117cdb8a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:51:46.712065 21889 ts_tablet_manager.cc:1905] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:46.712899 21844 tablet_bootstrap.cc:492] T b99d49335502422a80048e3626f92fb7 P e8f3c679f66046fdb39416b316bd6ab4: No bootstrap required, opened a new log
I20260619 14:51:46.712973 21844 ts_tablet_manager.cc:1397] T b99d49335502422a80048e3626f92fb7 P e8f3c679f66046fdb39416b316bd6ab4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260619 14:51:46.713408 21844 raft_consensus.cc:348] T b99d49335502422a80048e3626f92fb7 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:46.713518 21844 raft_consensus.cc:374] T b99d49335502422a80048e3626f92fb7 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:46.713557 21844 raft_consensus.cc:729] T b99d49335502422a80048e3626f92fb7 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f3c679f66046fdb39416b316bd6ab4, State: Initialized, Role: FOLLOWER
I20260619 14:51:46.713668 21844 consensus_queue.cc:260] T b99d49335502422a80048e3626f92fb7 P e8f3c679f66046fdb39416b316bd6ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:46.713935 21844 ts_tablet_manager.cc:1428] T b99d49335502422a80048e3626f92fb7 P e8f3c679f66046fdb39416b316bd6ab4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:51:46.714625 21889 ts_tablet_manager.cc:1918] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:51:46.714706 21889 log.cc:1196] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-1/wal/wals/484ef9af77184e738eaf98d1d38dc5a8
I20260619 14:51:46.714965 21889 ts_tablet_manager.cc:1939] T 484ef9af77184e738eaf98d1d38dc5a8 P 2f111686e0bf46fc801504db117cdb8a: Deleting consensus metadata
I20260619 14:51:46.715903 21321 catalog_manager.cc:4841] TS 2f111686e0bf46fc801504db117cdb8a (127.19.34.66:41937): tablet 484ef9af77184e738eaf98d1d38dc5a8 (table test-table [id=bb2f248ba9cc4e0883b4ca1fbdc9f57e]) successfully deleted
I20260619 14:51:47.009665 21851 raft_consensus.cc:482] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:51:47.009810 21851 raft_consensus.cc:504] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:47.010769 21851 leader_election.cc:288] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f111686e0bf46fc801504db117cdb8a (127.19.34.66:41937), e8f3c679f66046fdb39416b316bd6ab4 (127.19.34.67:46643)
I20260619 14:51:47.014245 21868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:47.011051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:46203 (local address 127.19.34.67:46643)
0619 14:51:47.011220 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:47.011225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:47.011390 (+ 165us) server_negotiation.cc:407] Connection header received
0619 14:51:47.011528 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:47.011531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:47.011566 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:47.011633 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:47.011873 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.012288 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.013259 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.013650 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.013782 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:47.013831 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:47.014031 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 14:51:47.014104 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 14:51:47.014266 21894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:47.010956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:46643 (local address 127.19.34.65:46203)
0619 14:51:47.011307 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:47.011316 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:47.011423 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:47.011651 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:47.011654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:47.011667 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:47.011774 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:47.011778 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.012429 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.012433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:47.013116 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:47.013123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.013787 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.013790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:47.013912 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:47.013928 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 14:51:47.014013 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 14:51:47.014088 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":18,"thread_start_us":94,"threads_started":1}
I20260619 14:51:47.014261 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:47.011019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:52921 (local address 127.19.34.66:41937)
0619 14:51:47.011279 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:47.011283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:47.011296 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:51:47.011428 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:47.011435 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:47.011473 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:47.011553 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:47.011801 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.012251 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.013225 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.013640 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.013771 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:47.013822 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:47.014051 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 14:51:47.014119 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":53,"threads_started":1}
I20260619 14:51:47.014282 21893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:47.010936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:41937 (local address 127.19.34.65:52921)
0619 14:51:47.011203 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:47.011217 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:47.011310 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:47.011563 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:47.011566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:47.011581 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:47.011702 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:47.011707 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.012384 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.012388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:47.013077 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:47.013085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.013760 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.013763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:47.013859 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:47.013945 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 14:51:47.014035 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 14:51:47.014111 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":58,"thread_start_us":76,"threads_started":1}
I20260619 14:51:47.014721 21774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99d49335502422a80048e3626f92fb7" candidate_uuid: "4d56944bba9847c4b319a72f051808ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8f3c679f66046fdb39416b316bd6ab4" is_pre_election: true
I20260619 14:51:47.014930 21774 raft_consensus.cc:2454] T b99d49335502422a80048e3626f92fb7 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d56944bba9847c4b319a72f051808ad in term 0.
I20260619 14:51:47.015007 21634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99d49335502422a80048e3626f92fb7" candidate_uuid: "4d56944bba9847c4b319a72f051808ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f111686e0bf46fc801504db117cdb8a" is_pre_election: true
I20260619 14:51:47.015120 21634 raft_consensus.cc:2454] T b99d49335502422a80048e3626f92fb7 P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d56944bba9847c4b319a72f051808ad in term 0.
I20260619 14:51:47.015312 21428 leader_election.cc:302] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d56944bba9847c4b319a72f051808ad, e8f3c679f66046fdb39416b316bd6ab4; no voters:
I20260619 14:51:47.015552 21851 raft_consensus.cc:2781] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:51:47.015607 21851 raft_consensus.cc:482] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:51:47.015633 21851 raft_consensus.cc:3037] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:47.016669 21851 raft_consensus.cc:504] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:47.017032 21851 leader_election.cc:288] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad [CANDIDATE]: Term 1 election: Requested vote from peers 2f111686e0bf46fc801504db117cdb8a (127.19.34.66:41937), e8f3c679f66046fdb39416b316bd6ab4 (127.19.34.67:46643)
I20260619 14:51:47.017338 21774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99d49335502422a80048e3626f92fb7" candidate_uuid: "4d56944bba9847c4b319a72f051808ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8f3c679f66046fdb39416b316bd6ab4"
I20260619 14:51:47.017406 21634 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b99d49335502422a80048e3626f92fb7" candidate_uuid: "4d56944bba9847c4b319a72f051808ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f111686e0bf46fc801504db117cdb8a"
I20260619 14:51:47.017448 21774 raft_consensus.cc:3037] T b99d49335502422a80048e3626f92fb7 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:47.017510 21634 raft_consensus.cc:3037] T b99d49335502422a80048e3626f92fb7 P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:47.018532 21774 raft_consensus.cc:2454] T b99d49335502422a80048e3626f92fb7 P e8f3c679f66046fdb39416b316bd6ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d56944bba9847c4b319a72f051808ad in term 1.
I20260619 14:51:47.018532 21634 raft_consensus.cc:2454] T b99d49335502422a80048e3626f92fb7 P 2f111686e0bf46fc801504db117cdb8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d56944bba9847c4b319a72f051808ad in term 1.
I20260619 14:51:47.018779 21428 leader_election.cc:302] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d56944bba9847c4b319a72f051808ad, e8f3c679f66046fdb39416b316bd6ab4; no voters:
I20260619 14:51:47.018932 21851 raft_consensus.cc:2781] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:47.019145 21851 raft_consensus.cc:686] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad [term 1 LEADER]: Becoming Leader. State: Replica: 4d56944bba9847c4b319a72f051808ad, State: Running, Role: LEADER
I20260619 14:51:47.019413 21851 consensus_queue.cc:237] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:47.021378 21335 catalog_manager.cc:5495] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d56944bba9847c4b319a72f051808ad (127.19.34.65). New cstate: current_term: 1 leader_uuid: "4d56944bba9847c4b319a72f051808ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } health_report { overall_health: UNKNOWN } } }
I20260619 14:51:47.079803 21898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:47.076578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.19.34.65:33029)
0619 14:51:47.076767 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:47.076773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:47.076785 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:51:47.076848 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:47.076850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:47.076901 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:47.077007 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:47.077207 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.077663 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.078430 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.078671 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.078775 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:47.079040 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:47.079456 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:47.079561 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:47.079621 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 14:51:47.079665 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":61,"threads_started":1}
I20260619 14:51:47.088745 21868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:47.085848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:55733 (local address 127.19.34.67:46643)
0619 14:51:47.085978 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:47.085982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:47.086118 (+ 136us) server_negotiation.cc:407] Connection header received
0619 14:51:47.086249 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:47.086252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:47.086294 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:47.086364 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:47.086580 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.087012 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.087783 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.087994 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.088095 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:47.088362 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:47.088428 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:47.088503 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:47.088589 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 14:51:47.088631 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:51:47.088769 21901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:47.085755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:46643 (local address 127.19.34.65:55733)
0619 14:51:47.086046 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:47.086056 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:47.086154 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:47.086374 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:47.086376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:47.086383 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:51:47.086476 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:47.086483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.087120 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.087125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:47.087671 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:47.087679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.088097 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.088100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:47.088192 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:47.088225 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:51:47.088508 (+ 283us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:51:47.088516 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:51:47.088572 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 14:51:47.088638 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":86,"threads_started":1}
I20260619 14:51:47.093753 21634 raft_consensus.cc:1261] T b99d49335502422a80048e3626f92fb7 P 2f111686e0bf46fc801504db117cdb8a [term 1 FOLLOWER]: Refusing update from remote peer 4d56944bba9847c4b319a72f051808ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:47.093793 21774 raft_consensus.cc:1261] T b99d49335502422a80048e3626f92fb7 P e8f3c679f66046fdb39416b316bd6ab4 [term 1 FOLLOWER]: Refusing update from remote peer 4d56944bba9847c4b319a72f051808ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:47.094194 21896 consensus_queue.cc:1035] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:51:47.094429 21851 consensus_queue.cc:1035] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:51:47.109181 21898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:47.106510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:59511 (local address 127.19.34.65:33029)
0619 14:51:47.106610 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:47.106614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:47.106626 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:51:47.106657 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:47.106659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:47.106700 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:47.106771 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:47.106971 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.107404 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.108150 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.108458 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.108548 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:47.108813 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:47.108880 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:47.108950 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:47.109032 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 14:51:47.109070 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 14:51:47.109184 21881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:47.106378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:33029 (local address 127.19.34.67:59511)
0619 14:51:47.106476 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:47.106491 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:47.106558 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:47.106779 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:47.106782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:47.106788 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:51:47.106869 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:47.106875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.107503 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.107508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:47.108039 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:47.108046 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:47.108558 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:47.108561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:47.108658 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:47.108709 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:51:47.108954 (+ 245us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:51:47.108959 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:51:47.109016 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 14:51:47.109089 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 14:51:47.109757 21870 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 14:51:48.660620 21336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:51:48.660806 21336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:51:48.662966 21336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de: Sending DeleteTablet for 3 replicas of tablet b99d49335502422a80048e3626f92fb7
I20260619 14:51:48.663511 21614 tablet_service.cc:1514] Processing DeleteTablet for tablet b99d49335502422a80048e3626f92fb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:48 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:39420
I20260619 14:51:48.663570 21474 tablet_service.cc:1514] Processing DeleteTablet for tablet b99d49335502422a80048e3626f92fb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:48 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:58172
I20260619 14:51:48.663678 21754 tablet_service.cc:1514] Processing DeleteTablet for tablet b99d49335502422a80048e3626f92fb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:48 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:49014
I20260619 14:51:48.663959 21927 tablet_replica.cc:330] T b99d49335502422a80048e3626f92fb7 P 2f111686e0bf46fc801504db117cdb8a: stopping tablet replica
I20260619 14:51:48.664094 21929 tablet_replica.cc:330] T b99d49335502422a80048e3626f92fb7 P e8f3c679f66046fdb39416b316bd6ab4: stopping tablet replica
I20260619 14:51:48.664143 21927 raft_consensus.cc:2229] T b99d49335502422a80048e3626f92fb7 P 2f111686e0bf46fc801504db117cdb8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:51:48.664335 21929 raft_consensus.cc:2229] T b99d49335502422a80048e3626f92fb7 P e8f3c679f66046fdb39416b316bd6ab4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:51:48.664354 21927 raft_consensus.cc:2258] T b99d49335502422a80048e3626f92fb7 P 2f111686e0bf46fc801504db117cdb8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:48.664461 21929 raft_consensus.cc:2258] T b99d49335502422a80048e3626f92fb7 P e8f3c679f66046fdb39416b316bd6ab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:48.664623 21928 tablet_replica.cc:330] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad: stopping tablet replica
I20260619 14:51:48.664757 21928 raft_consensus.cc:2229] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:51:48.665017 21928 raft_consensus.cc:2258] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:48.665230 21929 ts_tablet_manager.cc:1905] T b99d49335502422a80048e3626f92fb7 P e8f3c679f66046fdb39416b316bd6ab4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:48.665503 21927 ts_tablet_manager.cc:1905] T b99d49335502422a80048e3626f92fb7 P 2f111686e0bf46fc801504db117cdb8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:48.667687 21927 ts_tablet_manager.cc:1918] T b99d49335502422a80048e3626f92fb7 P 2f111686e0bf46fc801504db117cdb8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:51:48.667759 21927 log.cc:1196] T b99d49335502422a80048e3626f92fb7 P 2f111686e0bf46fc801504db117cdb8a: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-1/wal/wals/b99d49335502422a80048e3626f92fb7
I20260619 14:51:48.668051 21927 ts_tablet_manager.cc:1939] T b99d49335502422a80048e3626f92fb7 P 2f111686e0bf46fc801504db117cdb8a: Deleting consensus metadata
I20260619 14:51:48.668366 21929 ts_tablet_manager.cc:1918] T b99d49335502422a80048e3626f92fb7 P e8f3c679f66046fdb39416b316bd6ab4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:51:48.668503 21929 log.cc:1196] T b99d49335502422a80048e3626f92fb7 P e8f3c679f66046fdb39416b316bd6ab4: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/b99d49335502422a80048e3626f92fb7
I20260619 14:51:48.668713 21321 catalog_manager.cc:4841] TS 2f111686e0bf46fc801504db117cdb8a (127.19.34.66:41937): tablet b99d49335502422a80048e3626f92fb7 (table test-table [id=cc2c548bb4e2423b8188edeba8012bbf]) successfully deleted
I20260619 14:51:48.668769 21929 ts_tablet_manager.cc:1939] T b99d49335502422a80048e3626f92fb7 P e8f3c679f66046fdb39416b316bd6ab4: Deleting consensus metadata
I20260619 14:51:48.669029 21928 ts_tablet_manager.cc:1905] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:48.669364 21324 catalog_manager.cc:4841] TS e8f3c679f66046fdb39416b316bd6ab4 (127.19.34.67:46643): tablet b99d49335502422a80048e3626f92fb7 (table test-table [id=cc2c548bb4e2423b8188edeba8012bbf]) successfully deleted
I20260619 14:51:48.671353 21928 ts_tablet_manager.cc:1918] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:51:48.671439 21928 log.cc:1196] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-0/wal/wals/b99d49335502422a80048e3626f92fb7
I20260619 14:51:48.671733 21928 ts_tablet_manager.cc:1939] T b99d49335502422a80048e3626f92fb7 P 4d56944bba9847c4b319a72f051808ad: Deleting consensus metadata
I20260619 14:51:48.672475 21324 catalog_manager.cc:4841] TS 4d56944bba9847c4b319a72f051808ad (127.19.34.65:33029): tablet b99d49335502422a80048e3626f92fb7 (table test-table [id=cc2c548bb4e2423b8188edeba8012bbf]) successfully deleted
Jun 19 14:51:48 dist-test-slave-3qdh krb5kdc[21292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880708, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:51:48 dist-test-slave-3qdh krb5kdc[21292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880708, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:48.696653 21937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:48.682738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55562 (local address 127.19.34.126:33109)
0619 14:51:48.682997 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:48.683003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:48.683017 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:51:48.683139 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:48.683142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:48.683202 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:48.683284 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:48.683927 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.684561 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.685343 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.685560 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.685632 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:48.693631 (+ 7999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:48.693634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:48.693636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:48.693657 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:48.695029 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:48.695438 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:48.695442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:48.695444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:48.695523 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:48.695835 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:48.695838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:48.695839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:48.696014 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:48.696113 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:48.696422 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 14:51:48.696528 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":118,"threads_started":1}
I20260619 14:51:48.699148 21336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:51:48.699631 21336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:51:48.704074 21474 tablet_service.cc:1467] Processing CreateTablet for tablet 72adeea5b34b4d58906a88c029302502 (DEFAULT_TABLE table=test-table [id=feafce2dabdd424ebd1a866e648a3820]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:48.704241 21614 tablet_service.cc:1467] Processing CreateTablet for tablet 72adeea5b34b4d58906a88c029302502 (DEFAULT_TABLE table=test-table [id=feafce2dabdd424ebd1a866e648a3820]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:48.704406 21474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72adeea5b34b4d58906a88c029302502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:48.704394 21754 tablet_service.cc:1467] Processing CreateTablet for tablet 72adeea5b34b4d58906a88c029302502 (DEFAULT_TABLE table=test-table [id=feafce2dabdd424ebd1a866e648a3820]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:48.704541 21614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72adeea5b34b4d58906a88c029302502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:48.704645 21754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72adeea5b34b4d58906a88c029302502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:48.706488 21938 tablet_bootstrap.cc:492] T 72adeea5b34b4d58906a88c029302502 P 2f111686e0bf46fc801504db117cdb8a: Bootstrap starting.
I20260619 14:51:48.706947 21939 tablet_bootstrap.cc:492] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad: Bootstrap starting.
I20260619 14:51:48.707306 21940 tablet_bootstrap.cc:492] T 72adeea5b34b4d58906a88c029302502 P e8f3c679f66046fdb39416b316bd6ab4: Bootstrap starting.
I20260619 14:51:48.707856 21938 tablet_bootstrap.cc:654] T 72adeea5b34b4d58906a88c029302502 P 2f111686e0bf46fc801504db117cdb8a: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:48.707931 21939 tablet_bootstrap.cc:654] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:48.708232 21940 tablet_bootstrap.cc:654] T 72adeea5b34b4d58906a88c029302502 P e8f3c679f66046fdb39416b316bd6ab4: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:48.709110 21939 tablet_bootstrap.cc:492] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad: No bootstrap required, opened a new log
I20260619 14:51:48.709187 21939 ts_tablet_manager.cc:1397] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 14:51:48.709373 21940 tablet_bootstrap.cc:492] T 72adeea5b34b4d58906a88c029302502 P e8f3c679f66046fdb39416b316bd6ab4: No bootstrap required, opened a new log
I20260619 14:51:48.709465 21940 ts_tablet_manager.cc:1397] T 72adeea5b34b4d58906a88c029302502 P e8f3c679f66046fdb39416b316bd6ab4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:51:48.709647 21939 raft_consensus.cc:348] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:48.709759 21939 raft_consensus.cc:374] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:48.709805 21939 raft_consensus.cc:729] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d56944bba9847c4b319a72f051808ad, State: Initialized, Role: FOLLOWER
I20260619 14:51:48.709875 21940 raft_consensus.cc:348] T 72adeea5b34b4d58906a88c029302502 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:48.709924 21939 consensus_queue.cc:260] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:48.709976 21940 raft_consensus.cc:374] T 72adeea5b34b4d58906a88c029302502 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:48.710007 21940 raft_consensus.cc:729] T 72adeea5b34b4d58906a88c029302502 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f3c679f66046fdb39416b316bd6ab4, State: Initialized, Role: FOLLOWER
I20260619 14:51:48.710148 21940 consensus_queue.cc:260] T 72adeea5b34b4d58906a88c029302502 P e8f3c679f66046fdb39416b316bd6ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:48.710239 21939 ts_tablet_manager.cc:1428] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 14:51:48.710414 21940 ts_tablet_manager.cc:1428] T 72adeea5b34b4d58906a88c029302502 P e8f3c679f66046fdb39416b316bd6ab4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:51:48.710813 21938 tablet_bootstrap.cc:492] T 72adeea5b34b4d58906a88c029302502 P 2f111686e0bf46fc801504db117cdb8a: No bootstrap required, opened a new log
I20260619 14:51:48.710915 21938 ts_tablet_manager.cc:1397] T 72adeea5b34b4d58906a88c029302502 P 2f111686e0bf46fc801504db117cdb8a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 14:51:48.711369 21938 raft_consensus.cc:348] T 72adeea5b34b4d58906a88c029302502 P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:48.711534 21938 raft_consensus.cc:374] T 72adeea5b34b4d58906a88c029302502 P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:48.711566 21938 raft_consensus.cc:729] T 72adeea5b34b4d58906a88c029302502 P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f111686e0bf46fc801504db117cdb8a, State: Initialized, Role: FOLLOWER
I20260619 14:51:48.711755 21938 consensus_queue.cc:260] T 72adeea5b34b4d58906a88c029302502 P 2f111686e0bf46fc801504db117cdb8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:48.712201 21938 ts_tablet_manager.cc:1428] T 72adeea5b34b4d58906a88c029302502 P 2f111686e0bf46fc801504db117cdb8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:51:48.814596 21918 raft_consensus.cc:482] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:51:48.814766 21918 raft_consensus.cc:504] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:48.815289 21918 leader_election.cc:288] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f111686e0bf46fc801504db117cdb8a (127.19.34.66:41937), e8f3c679f66046fdb39416b316bd6ab4 (127.19.34.67:46643)
I20260619 14:51:48.815743 21774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72adeea5b34b4d58906a88c029302502" candidate_uuid: "4d56944bba9847c4b319a72f051808ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8f3c679f66046fdb39416b316bd6ab4" is_pre_election: true
I20260619 14:51:48.815919 21774 raft_consensus.cc:2454] T 72adeea5b34b4d58906a88c029302502 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d56944bba9847c4b319a72f051808ad in term 0.
I20260619 14:51:48.816511 21428 leader_election.cc:302] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d56944bba9847c4b319a72f051808ad, e8f3c679f66046fdb39416b316bd6ab4; no voters:
I20260619 14:51:48.816762 21918 raft_consensus.cc:2781] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:51:48.816736 21633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72adeea5b34b4d58906a88c029302502" candidate_uuid: "4d56944bba9847c4b319a72f051808ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f111686e0bf46fc801504db117cdb8a" is_pre_election: true
I20260619 14:51:48.816834 21918 raft_consensus.cc:482] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:51:48.816885 21918 raft_consensus.cc:3037] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:48.816912 21633 raft_consensus.cc:2454] T 72adeea5b34b4d58906a88c029302502 P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d56944bba9847c4b319a72f051808ad in term 0.
I20260619 14:51:48.818159 21918 raft_consensus.cc:504] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:48.818572 21918 leader_election.cc:288] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad [CANDIDATE]: Term 1 election: Requested vote from peers 2f111686e0bf46fc801504db117cdb8a (127.19.34.66:41937), e8f3c679f66046fdb39416b316bd6ab4 (127.19.34.67:46643)
I20260619 14:51:48.818897 21633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72adeea5b34b4d58906a88c029302502" candidate_uuid: "4d56944bba9847c4b319a72f051808ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f111686e0bf46fc801504db117cdb8a"
I20260619 14:51:48.819031 21633 raft_consensus.cc:3037] T 72adeea5b34b4d58906a88c029302502 P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:48.819427 21774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72adeea5b34b4d58906a88c029302502" candidate_uuid: "4d56944bba9847c4b319a72f051808ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8f3c679f66046fdb39416b316bd6ab4"
I20260619 14:51:48.819542 21774 raft_consensus.cc:3037] T 72adeea5b34b4d58906a88c029302502 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:48.820384 21633 raft_consensus.cc:2454] T 72adeea5b34b4d58906a88c029302502 P 2f111686e0bf46fc801504db117cdb8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d56944bba9847c4b319a72f051808ad in term 1.
I20260619 14:51:48.820739 21425 leader_election.cc:302] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f111686e0bf46fc801504db117cdb8a, 4d56944bba9847c4b319a72f051808ad; no voters:
I20260619 14:51:48.820979 21774 raft_consensus.cc:2454] T 72adeea5b34b4d58906a88c029302502 P e8f3c679f66046fdb39416b316bd6ab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d56944bba9847c4b319a72f051808ad in term 1.
I20260619 14:51:48.821218 21918 raft_consensus.cc:2781] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:48.821302 21918 raft_consensus.cc:686] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad [term 1 LEADER]: Becoming Leader. State: Replica: 4d56944bba9847c4b319a72f051808ad, State: Running, Role: LEADER
I20260619 14:51:48.821470 21918 consensus_queue.cc:237] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:48.823681 21337 catalog_manager.cc:5495] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d56944bba9847c4b319a72f051808ad (127.19.34.65). New cstate: current_term: 1 leader_uuid: "4d56944bba9847c4b319a72f051808ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } health_report { overall_health: UNKNOWN } } }
I20260619 14:51:48.869647 21947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:48.866643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35116 (local address 127.19.34.65:33029)
0619 14:51:48.866911 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:48.866925 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:48.866938 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:51:48.866977 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:48.866979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:48.867019 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:48.867161 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:48.867386 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.867895 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.868748 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:48.868954 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:48.869035 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:48.869235 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:48.869299 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:48.869363 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:48.869432 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 14:51:48.869471 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":127,"threads_started":1}
I20260619 14:51:48.875039 21774 raft_consensus.cc:1261] T 72adeea5b34b4d58906a88c029302502 P e8f3c679f66046fdb39416b316bd6ab4 [term 1 FOLLOWER]: Refusing update from remote peer 4d56944bba9847c4b319a72f051808ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:48.875073 21633 raft_consensus.cc:1261] T 72adeea5b34b4d58906a88c029302502 P 2f111686e0bf46fc801504db117cdb8a [term 1 FOLLOWER]: Refusing update from remote peer 4d56944bba9847c4b319a72f051808ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:48.875422 21922 consensus_queue.cc:1035] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:51:48.875568 21918 consensus_queue.cc:1035] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:51:48.903481 21337 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:51:48.903609 21337 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:51:48.905424 21337 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de: Sending DeleteTablet for 3 replicas of tablet 72adeea5b34b4d58906a88c029302502
I20260619 14:51:48.905884 21614 tablet_service.cc:1514] Processing DeleteTablet for tablet 72adeea5b34b4d58906a88c029302502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:48 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:39420
I20260619 14:51:48.905975 21754 tablet_service.cc:1514] Processing DeleteTablet for tablet 72adeea5b34b4d58906a88c029302502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:48 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:49014
I20260619 14:51:48.906073 21474 tablet_service.cc:1514] Processing DeleteTablet for tablet 72adeea5b34b4d58906a88c029302502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:48 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:58172
I20260619 14:51:48.906147 21929 tablet_replica.cc:330] T 72adeea5b34b4d58906a88c029302502 P e8f3c679f66046fdb39416b316bd6ab4: stopping tablet replica
I20260619 14:51:48.906246 21927 tablet_replica.cc:330] T 72adeea5b34b4d58906a88c029302502 P 2f111686e0bf46fc801504db117cdb8a: stopping tablet replica
I20260619 14:51:48.906277 21929 raft_consensus.cc:2229] T 72adeea5b34b4d58906a88c029302502 P e8f3c679f66046fdb39416b316bd6ab4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:51:48.906368 21927 raft_consensus.cc:2229] T 72adeea5b34b4d58906a88c029302502 P 2f111686e0bf46fc801504db117cdb8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:51:48.906383 21929 raft_consensus.cc:2258] T 72adeea5b34b4d58906a88c029302502 P e8f3c679f66046fdb39416b316bd6ab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:48.906486 21927 raft_consensus.cc:2258] T 72adeea5b34b4d58906a88c029302502 P 2f111686e0bf46fc801504db117cdb8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:48.906630 21928 tablet_replica.cc:330] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad: stopping tablet replica
I20260619 14:51:48.906746 21928 raft_consensus.cc:2229] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:51:48.906899 21928 raft_consensus.cc:2258] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:48.907105 21929 ts_tablet_manager.cc:1905] T 72adeea5b34b4d58906a88c029302502 P e8f3c679f66046fdb39416b316bd6ab4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:48.907398 21927 ts_tablet_manager.cc:1905] T 72adeea5b34b4d58906a88c029302502 P 2f111686e0bf46fc801504db117cdb8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:48.907393 21928 ts_tablet_manager.cc:1905] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:48.909245 21929 ts_tablet_manager.cc:1918] T 72adeea5b34b4d58906a88c029302502 P e8f3c679f66046fdb39416b316bd6ab4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:51:48.909329 21929 log.cc:1196] T 72adeea5b34b4d58906a88c029302502 P e8f3c679f66046fdb39416b316bd6ab4: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/72adeea5b34b4d58906a88c029302502
I20260619 14:51:48.909526 21927 ts_tablet_manager.cc:1918] T 72adeea5b34b4d58906a88c029302502 P 2f111686e0bf46fc801504db117cdb8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:51:48.909526 21928 ts_tablet_manager.cc:1918] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:51:48.909595 21928 log.cc:1196] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-0/wal/wals/72adeea5b34b4d58906a88c029302502
I20260619 14:51:48.909595 21927 log.cc:1196] T 72adeea5b34b4d58906a88c029302502 P 2f111686e0bf46fc801504db117cdb8a: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-1/wal/wals/72adeea5b34b4d58906a88c029302502
I20260619 14:51:48.909602 21929 ts_tablet_manager.cc:1939] T 72adeea5b34b4d58906a88c029302502 P e8f3c679f66046fdb39416b316bd6ab4: Deleting consensus metadata
I20260619 14:51:48.909849 21928 ts_tablet_manager.cc:1939] T 72adeea5b34b4d58906a88c029302502 P 4d56944bba9847c4b319a72f051808ad: Deleting consensus metadata
I20260619 14:51:48.909849 21927 ts_tablet_manager.cc:1939] T 72adeea5b34b4d58906a88c029302502 P 2f111686e0bf46fc801504db117cdb8a: Deleting consensus metadata
I20260619 14:51:48.910311 21337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 14:51:48.910468 21324 catalog_manager.cc:4841] TS e8f3c679f66046fdb39416b316bd6ab4 (127.19.34.67:46643): tablet 72adeea5b34b4d58906a88c029302502 (table test-table [id=feafce2dabdd424ebd1a866e648a3820]) successfully deleted
I20260619 14:51:48.910473 21321 catalog_manager.cc:4841] TS 2f111686e0bf46fc801504db117cdb8a (127.19.34.66:41937): tablet 72adeea5b34b4d58906a88c029302502 (table test-table [id=feafce2dabdd424ebd1a866e648a3820]) successfully deleted
I20260619 14:51:48.910645 21324 catalog_manager.cc:4841] TS 4d56944bba9847c4b319a72f051808ad (127.19.34.65:33029): tablet 72adeea5b34b4d58906a88c029302502 (table test-table [id=feafce2dabdd424ebd1a866e648a3820]) successfully deleted
W20260619 14:51:48.910779 21337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:51:48.914798 21474 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf00fac8ba742c396bed0686c1722c6 (DEFAULT_TABLE table=test-table [id=e125c5c751bb48c49642a83d8f46e5f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:48.914883 21614 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf00fac8ba742c396bed0686c1722c6 (DEFAULT_TABLE table=test-table [id=e125c5c751bb48c49642a83d8f46e5f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:48.915011 21474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf00fac8ba742c396bed0686c1722c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:48.915149 21614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf00fac8ba742c396bed0686c1722c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:48.916416 21939 tablet_bootstrap.cc:492] T 6bf00fac8ba742c396bed0686c1722c6 P 4d56944bba9847c4b319a72f051808ad: Bootstrap starting.
I20260619 14:51:48.916822 21938 tablet_bootstrap.cc:492] T 6bf00fac8ba742c396bed0686c1722c6 P 2f111686e0bf46fc801504db117cdb8a: Bootstrap starting.
I20260619 14:51:48.916937 21754 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf00fac8ba742c396bed0686c1722c6 (DEFAULT_TABLE table=test-table [id=e125c5c751bb48c49642a83d8f46e5f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:48.917138 21754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf00fac8ba742c396bed0686c1722c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:48.917147 21939 tablet_bootstrap.cc:654] T 6bf00fac8ba742c396bed0686c1722c6 P 4d56944bba9847c4b319a72f051808ad: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:48.917485 21938 tablet_bootstrap.cc:654] T 6bf00fac8ba742c396bed0686c1722c6 P 2f111686e0bf46fc801504db117cdb8a: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:48.918169 21939 tablet_bootstrap.cc:492] T 6bf00fac8ba742c396bed0686c1722c6 P 4d56944bba9847c4b319a72f051808ad: No bootstrap required, opened a new log
I20260619 14:51:48.918264 21939 ts_tablet_manager.cc:1397] T 6bf00fac8ba742c396bed0686c1722c6 P 4d56944bba9847c4b319a72f051808ad: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 14:51:48.918608 21939 raft_consensus.cc:348] T 6bf00fac8ba742c396bed0686c1722c6 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:48.918705 21939 raft_consensus.cc:374] T 6bf00fac8ba742c396bed0686c1722c6 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:48.918741 21939 raft_consensus.cc:729] T 6bf00fac8ba742c396bed0686c1722c6 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d56944bba9847c4b319a72f051808ad, State: Initialized, Role: FOLLOWER
I20260619 14:51:48.918852 21939 consensus_queue.cc:260] T 6bf00fac8ba742c396bed0686c1722c6 P 4d56944bba9847c4b319a72f051808ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:48.919097 21939 ts_tablet_manager.cc:1428] T 6bf00fac8ba742c396bed0686c1722c6 P 4d56944bba9847c4b319a72f051808ad: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 14:51:48.919167 21940 tablet_bootstrap.cc:492] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4: Bootstrap starting.
I20260619 14:51:48.919821 21938 tablet_bootstrap.cc:492] T 6bf00fac8ba742c396bed0686c1722c6 P 2f111686e0bf46fc801504db117cdb8a: No bootstrap required, opened a new log
I20260619 14:51:48.919881 21938 ts_tablet_manager.cc:1397] T 6bf00fac8ba742c396bed0686c1722c6 P 2f111686e0bf46fc801504db117cdb8a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 14:51:48.919996 21940 tablet_bootstrap.cc:654] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:48.920241 21938 raft_consensus.cc:348] T 6bf00fac8ba742c396bed0686c1722c6 P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:48.920373 21938 raft_consensus.cc:374] T 6bf00fac8ba742c396bed0686c1722c6 P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:48.920424 21938 raft_consensus.cc:729] T 6bf00fac8ba742c396bed0686c1722c6 P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f111686e0bf46fc801504db117cdb8a, State: Initialized, Role: FOLLOWER
I20260619 14:51:48.920536 21938 consensus_queue.cc:260] T 6bf00fac8ba742c396bed0686c1722c6 P 2f111686e0bf46fc801504db117cdb8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:48.920742 21938 ts_tablet_manager.cc:1428] T 6bf00fac8ba742c396bed0686c1722c6 P 2f111686e0bf46fc801504db117cdb8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:51:48.921680 21940 tablet_bootstrap.cc:492] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4: No bootstrap required, opened a new log
I20260619 14:51:48.921792 21940 ts_tablet_manager.cc:1397] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 14:51:48.922210 21940 raft_consensus.cc:348] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:48.922339 21940 raft_consensus.cc:374] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:48.922376 21940 raft_consensus.cc:729] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f3c679f66046fdb39416b316bd6ab4, State: Initialized, Role: FOLLOWER
I20260619 14:51:48.922495 21940 consensus_queue.cc:260] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:48.922781 21940 ts_tablet_manager.cc:1428] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:51:48.975312 21951 raft_consensus.cc:482] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:51:48.975418 21951 raft_consensus.cc:504] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:48.975747 21951 leader_election.cc:288] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d56944bba9847c4b319a72f051808ad (127.19.34.65:33029), 2f111686e0bf46fc801504db117cdb8a (127.19.34.66:41937)
I20260619 14:51:48.976092 21494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf00fac8ba742c396bed0686c1722c6" candidate_uuid: "e8f3c679f66046fdb39416b316bd6ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d56944bba9847c4b319a72f051808ad" is_pre_election: true
I20260619 14:51:48.976150 21633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf00fac8ba742c396bed0686c1722c6" candidate_uuid: "e8f3c679f66046fdb39416b316bd6ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f111686e0bf46fc801504db117cdb8a" is_pre_election: true
I20260619 14:51:48.976269 21494 raft_consensus.cc:2454] T 6bf00fac8ba742c396bed0686c1722c6 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f3c679f66046fdb39416b316bd6ab4 in term 0.
I20260619 14:51:48.976291 21633 raft_consensus.cc:2454] T 6bf00fac8ba742c396bed0686c1722c6 P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8f3c679f66046fdb39416b316bd6ab4 in term 0.
I20260619 14:51:48.976565 21705 leader_election.cc:302] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f111686e0bf46fc801504db117cdb8a, e8f3c679f66046fdb39416b316bd6ab4; no voters:
I20260619 14:51:48.976737 21951 raft_consensus.cc:2781] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:51:48.976795 21951 raft_consensus.cc:482] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:51:48.976850 21951 raft_consensus.cc:3037] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:48.977682 21951 raft_consensus.cc:504] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:48.977973 21951 leader_election.cc:288] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 [CANDIDATE]: Term 1 election: Requested vote from peers 4d56944bba9847c4b319a72f051808ad (127.19.34.65:33029), 2f111686e0bf46fc801504db117cdb8a (127.19.34.66:41937)
I20260619 14:51:48.978273 21633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf00fac8ba742c396bed0686c1722c6" candidate_uuid: "e8f3c679f66046fdb39416b316bd6ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f111686e0bf46fc801504db117cdb8a"
I20260619 14:51:48.978284 21494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf00fac8ba742c396bed0686c1722c6" candidate_uuid: "e8f3c679f66046fdb39416b316bd6ab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d56944bba9847c4b319a72f051808ad"
I20260619 14:51:48.978387 21633 raft_consensus.cc:3037] T 6bf00fac8ba742c396bed0686c1722c6 P 2f111686e0bf46fc801504db117cdb8a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:48.978391 21494 raft_consensus.cc:3037] T 6bf00fac8ba742c396bed0686c1722c6 P 4d56944bba9847c4b319a72f051808ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:48.979455 21633 raft_consensus.cc:2454] T 6bf00fac8ba742c396bed0686c1722c6 P 2f111686e0bf46fc801504db117cdb8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f3c679f66046fdb39416b316bd6ab4 in term 1.
I20260619 14:51:48.979461 21494 raft_consensus.cc:2454] T 6bf00fac8ba742c396bed0686c1722c6 P 4d56944bba9847c4b319a72f051808ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8f3c679f66046fdb39416b316bd6ab4 in term 1.
I20260619 14:51:48.979730 21705 leader_election.cc:302] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f111686e0bf46fc801504db117cdb8a, e8f3c679f66046fdb39416b316bd6ab4; no voters:
I20260619 14:51:48.979943 21951 raft_consensus.cc:2781] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:48.980049 21951 raft_consensus.cc:686] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 [term 1 LEADER]: Becoming Leader. State: Replica: e8f3c679f66046fdb39416b316bd6ab4, State: Running, Role: LEADER
I20260619 14:51:48.980203 21951 consensus_queue.cc:237] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } }
I20260619 14:51:48.981769 21335 catalog_manager.cc:5495] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8f3c679f66046fdb39416b316bd6ab4 (127.19.34.67). New cstate: current_term: 1 leader_uuid: "e8f3c679f66046fdb39416b316bd6ab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8f3c679f66046fdb39416b316bd6ab4" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 46643 } health_report { overall_health: HEALTHY } } }
I20260619 14:51:49.004657 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:49.001791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42866 (local address 127.19.34.67:46643)
0619 14:51:49.002052 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:49.002058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:49.002070 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:51:49.002105 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:49.002107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:49.002156 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:49.002227 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:49.002416 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:49.002902 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:49.003673 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:49.003893 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:49.003976 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:49.004173 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:49.004236 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:49.004290 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:49.004428 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 14:51:49.004486 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":114,"threads_started":1}
I20260619 14:51:49.009706 21633 raft_consensus.cc:1261] T 6bf00fac8ba742c396bed0686c1722c6 P 2f111686e0bf46fc801504db117cdb8a [term 1 FOLLOWER]: Refusing update from remote peer e8f3c679f66046fdb39416b316bd6ab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:49.009732 21494 raft_consensus.cc:1261] T 6bf00fac8ba742c396bed0686c1722c6 P 4d56944bba9847c4b319a72f051808ad [term 1 FOLLOWER]: Refusing update from remote peer e8f3c679f66046fdb39416b316bd6ab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:49.010051 21913 consensus_queue.cc:1035] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f111686e0bf46fc801504db117cdb8a" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:51:49.010254 21886 consensus_queue.cc:1035] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d56944bba9847c4b319a72f051808ad" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:51:49.021695 21906 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 14:51:50.566994 21335 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:51:50.567200 21335 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:51:50.569541 21335 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58d8ade3b66a41e5bb311f4f347e30de: Sending DeleteTablet for 3 replicas of tablet 6bf00fac8ba742c396bed0686c1722c6
I20260619 14:51:50.570003 21474 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf00fac8ba742c396bed0686c1722c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:50 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:58172
I20260619 14:51:50.570096 21754 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf00fac8ba742c396bed0686c1722c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:50 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:49014
I20260619 14:51:50.570339 21614 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf00fac8ba742c396bed0686c1722c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:51:50 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:39420
I20260619 14:51:50.570415 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 21409
I20260619 14:51:50.570344 21979 tablet_replica.cc:330] T 6bf00fac8ba742c396bed0686c1722c6 P 4d56944bba9847c4b319a72f051808ad: stopping tablet replica
I20260619 14:51:50.571419 21981 tablet_replica.cc:330] T 6bf00fac8ba742c396bed0686c1722c6 P 2f111686e0bf46fc801504db117cdb8a: stopping tablet replica
I20260619 14:51:50.571581 21981 raft_consensus.cc:2229] T 6bf00fac8ba742c396bed0686c1722c6 P 2f111686e0bf46fc801504db117cdb8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:51:50.571731 21981 raft_consensus.cc:2258] T 6bf00fac8ba742c396bed0686c1722c6 P 2f111686e0bf46fc801504db117cdb8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:50.572449 21981 ts_tablet_manager.cc:1905] T 6bf00fac8ba742c396bed0686c1722c6 P 2f111686e0bf46fc801504db117cdb8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:51:50.574759 21981 ts_tablet_manager.cc:1918] T 6bf00fac8ba742c396bed0686c1722c6 P 2f111686e0bf46fc801504db117cdb8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:51:50.574828 21981 log.cc:1196] T 6bf00fac8ba742c396bed0686c1722c6 P 2f111686e0bf46fc801504db117cdb8a: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-1/wal/wals/6bf00fac8ba742c396bed0686c1722c6
I20260619 14:51:50.575098 21981 ts_tablet_manager.cc:1939] T 6bf00fac8ba742c396bed0686c1722c6 P 2f111686e0bf46fc801504db117cdb8a: Deleting consensus metadata
I20260619 14:51:50.575730 21321 catalog_manager.cc:4841] TS 2f111686e0bf46fc801504db117cdb8a (127.19.34.66:41937): tablet 6bf00fac8ba742c396bed0686c1722c6 (table test-table [id=e125c5c751bb48c49642a83d8f46e5f8]) successfully deleted
I20260619 14:51:50.576459 21980 tablet_replica.cc:330] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4: stopping tablet replica
I20260619 14:51:50.576608 21980 raft_consensus.cc:2229] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:51:50.576788 21980 raft_consensus.cc:2258] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:51:50.577368 21980 ts_tablet_manager.cc:1905] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 14:51:50.578188 21324 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.34.65:33029 (error 108)
I20260619 14:51:50.578719 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 21549
I20260619 14:51:50.583525 21980 ts_tablet_manager.cc:1918] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:51:50.583609 21980 log.cc:1196] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/6bf00fac8ba742c396bed0686c1722c6
I20260619 14:51:50.583881 21980 ts_tablet_manager.cc:1939] T 6bf00fac8ba742c396bed0686c1722c6 P e8f3c679f66046fdb39416b316bd6ab4: Deleting consensus metadata
I20260619 14:51:50.584107 21983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:50.578922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:33029 (local address 127.0.0.1:35126)
0619 14:51:50.583859 (+ 4937us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:50.583981 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.65:33029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4827,"thread_start_us":76,"threads_started":1}
W20260619 14:51:50.584234 21324 catalog_manager.cc:4584] TS 4d56944bba9847c4b319a72f051808ad (127.19.34.65:33029): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6bf00fac8ba742c396bed0686c1722c6: Network error: Client connection negotiation failed: client connection to 127.19.34.65:33029: connect: Connection refused (error 111)
I20260619 14:51:50.584776 21324 catalog_manager.cc:4841] TS e8f3c679f66046fdb39416b316bd6ab4 (127.19.34.67:46643): tablet 6bf00fac8ba742c396bed0686c1722c6 (table test-table [id=e125c5c751bb48c49642a83d8f46e5f8]) successfully deleted
I20260619 14:51:50.585683 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 21689
I20260619 14:51:50.595019 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 21305
2026-06-19T14:51:50Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6514 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:50 dist-test-slave-3qdh krb5kdc[21986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:50 dist-test-slave-3qdh krb5kdc[21986](info): set up 2 sockets
Jun 19 14:51:50 dist-test-slave-3qdh krb5kdc[21986](info): commencing operation
krb5kdc: starting...
W20260619 14:51:51.563936 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:51:51 dist-test-slave-3qdh krb5kdc[21986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:51:51.671061 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:44413
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:44413
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:51.770056 21999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:51.770293 21999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:51.770357 21999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:51.773856 21999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:51.773931 21999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:51.773967 21999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:51.774030 21999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:51.774070 21999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:51.778528 21999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:44413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:44413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:51.779321 21999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:51.780184 21999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:51.788010 22004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:51.788031 22005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:51.788014 22007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:51.788058 21999 server_base.cc:1034] running on GCE node
I20260619 14:51:51.788699 21999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:51.789525 21999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:51.790673 21999 hybrid_clock.cc:648] HybridClock initialized: now 1781880711790650 us; error 34 us; skew 500 ppm
Jun 19 14:51:51 dist-test-slave-3qdh krb5kdc[21986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880711, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:51.792973 21999 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:51:51.793682 21999 webserver.cc:460] Webserver started at http://127.19.34.126:43777/ using document root <none> and password file <none>
I20260619 14:51:51.794170 21999 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:51.794229 21999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:51.794445 21999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:51.796185 21999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "611e103962744671b83e48c3742b9ebe"
format_stamp: "Formatted at 2026-06-19 14:51:51 on dist-test-slave-3qdh"
server_key: "3a24164c4b4394e6711dd1d5bd2770a4"
server_key_iv: "a1e009ec9aeeeefa5983ab399c793bff"
server_key_version: "encryptionkey@0"
I20260619 14:51:51.796752 21999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "611e103962744671b83e48c3742b9ebe"
format_stamp: "Formatted at 2026-06-19 14:51:51 on dist-test-slave-3qdh"
server_key: "3a24164c4b4394e6711dd1d5bd2770a4"
server_key_iv: "a1e009ec9aeeeefa5983ab399c793bff"
server_key_version: "encryptionkey@0"
I20260619 14:51:51.799837 21999 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:51:51.801895 22014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:51.802907 21999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:51:51.803015 21999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "611e103962744671b83e48c3742b9ebe"
format_stamp: "Formatted at 2026-06-19 14:51:51 on dist-test-slave-3qdh"
server_key: "3a24164c4b4394e6711dd1d5bd2770a4"
server_key_iv: "a1e009ec9aeeeefa5983ab399c793bff"
server_key_version: "encryptionkey@0"
I20260619 14:51:51.803100 21999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:51.822069 21999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:51.824510 21999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:51.824708 21999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:51.830621 21999 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:44413
I20260619 14:51:51.830638 22066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:44413 every 8 connection(s)
I20260619 14:51:51.831403 21999 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:51.833809 22067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:51.836015 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 21999
I20260619 14:51:51.836122 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:51:51.836388 19593 external_mini_cluster.cc:1428] Setting key 100e3c666169becc5b37fbff970d5a8e
I20260619 14:51:51.839087 22067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe: Bootstrap starting.
I20260619 14:51:51.841464 22067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:51.842149 22067 log.cc:824] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:51.843842 22067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe: No bootstrap required, opened a new log
Jun 19 14:51:51 dist-test-slave-3qdh krb5kdc[21986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:51.847177 22067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611e103962744671b83e48c3742b9ebe" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44413 } }
I20260619 14:51:51.847359 22067 raft_consensus.cc:374] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:51.847431 22067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 611e103962744671b83e48c3742b9ebe, State: Initialized, Role: FOLLOWER
I20260619 14:51:51.847841 22067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611e103962744671b83e48c3742b9ebe" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44413 } }
I20260619 14:51:51.847981 22067 raft_consensus.cc:388] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:51.848045 22067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:51.848143 22067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:51.849042 22067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611e103962744671b83e48c3742b9ebe" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44413 } }
I20260619 14:51:51.849370 22067 leader_election.cc:302] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 611e103962744671b83e48c3742b9ebe; no voters:
I20260619 14:51:51.849647 22067 leader_election.cc:288] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:51.849716 22072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:51.849865 22072 raft_consensus.cc:686] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe [term 1 LEADER]: Becoming Leader. State: Replica: 611e103962744671b83e48c3742b9ebe, State: Running, Role: LEADER
I20260619 14:51:51.850062 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:51.837524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38936 (local address 127.19.34.126:44413)
0619 14:51:51.837895 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:51.837911 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:51.837947 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:51:51.838651 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:51.838669 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:51.839011 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:51.839313 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:51.839915 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:51.841342 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:51.842177 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:51.842686 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:51.842864 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:51.845326 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:51.845331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:51.845342 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:51.845365 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:51.847362 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:51.847782 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:51.847787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:51.847792 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:51.847903 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:51.848214 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:51.848219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:51.848220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:51.848603 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:51.848746 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:51.849010 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 14:51:51.849274 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":89,"threads_started":1}
I20260619 14:51:51.850170 22072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611e103962744671b83e48c3742b9ebe" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44413 } }
I20260619 14:51:51.850685 22067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:51.851608 22074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 611e103962744671b83e48c3742b9ebe. Latest consensus state: current_term: 1 leader_uuid: "611e103962744671b83e48c3742b9ebe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611e103962744671b83e48c3742b9ebe" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44413 } } }
I20260619 14:51:51.851792 22074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:51.852728 22073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "611e103962744671b83e48c3742b9ebe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611e103962744671b83e48c3742b9ebe" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44413 } } }
I20260619 14:51:51.853025 22073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe [sys.catalog]: This master's current role is: LEADER
W20260619 14:51:51.854871 22087 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:51:51.854938 22087 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:51:51.855003 22081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:51.855684 22081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:51.860033 22081 catalog_manager.cc:1347] Generated new cluster ID: e72af8f500a0415a86f0fffa6fe806d9
I20260619 14:51:51.860086 22081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:51.887915 22081 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:51.888865 22081 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:51.899467 22081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 611e103962744671b83e48c3742b9ebe: Generated new TSK 0
I20260619 14:51:51.900079 22081 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.34.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
I20260619 14:51:51.926788 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44413
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:52.027653 22093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:52.027933 22093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:52.027976 22093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:52.031652 22093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:52.031701 22093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:52.031754 22093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:51:52.036324 22093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:52.037050 22093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:52.037827 22093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:52.046358 22099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:52.046360 22101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:52.046404 22093 server_base.cc:1034] running on GCE node
W20260619 14:51:52.046372 22098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:52.047020 22093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:52.047585 22093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:52.048763 22093 hybrid_clock.cc:648] HybridClock initialized: now 1781880712048751 us; error 30 us; skew 500 ppm
Jun 19 14:51:52 dist-test-slave-3qdh krb5kdc[21986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880712, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:52.051347 22093 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:51:52.052134 22093 webserver.cc:460] Webserver started at http://127.19.34.65:43473/ using document root <none> and password file <none>
I20260619 14:51:52.052765 22093 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:52.052860 22093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:52.053076 22093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:52.054919 22093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "2c1586c3e7fb4275a5cde6ae84cc759e"
format_stamp: "Formatted at 2026-06-19 14:51:52 on dist-test-slave-3qdh"
server_key: "324afc874e5ba31206c85b61097f4546"
server_key_iv: "e291f61d0e39a8e93f2c40cb965c28a5"
server_key_version: "encryptionkey@0"
I20260619 14:51:52.055474 22093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "2c1586c3e7fb4275a5cde6ae84cc759e"
format_stamp: "Formatted at 2026-06-19 14:51:52 on dist-test-slave-3qdh"
server_key: "324afc874e5ba31206c85b61097f4546"
server_key_iv: "e291f61d0e39a8e93f2c40cb965c28a5"
server_key_version: "encryptionkey@0"
I20260619 14:51:52.058652 22093 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:51:52.060738 22108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:52.061856 22093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:52.061990 22093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "2c1586c3e7fb4275a5cde6ae84cc759e"
format_stamp: "Formatted at 2026-06-19 14:51:52 on dist-test-slave-3qdh"
server_key: "324afc874e5ba31206c85b61097f4546"
server_key_iv: "e291f61d0e39a8e93f2c40cb965c28a5"
server_key_version: "encryptionkey@0"
I20260619 14:51:52.062088 22093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:52.075572 22093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:52.078087 22093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:52.078277 22093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:52.078758 22093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:52.079465 22093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:52.079536 22093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:52.079603 22093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:52.079656 22093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:52.089119 22093 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:46477
I20260619 14:51:52.089489 22221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:46477 every 8 connection(s)
I20260619 14:51:52.090088 22093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:51:52.091702 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 22093
I20260619 14:51:52.091818 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:51:52.092092 19593 external_mini_cluster.cc:1428] Setting key 1860d6ad647189382ce2714b23556f6c
Jun 19 14:51:52 dist-test-slave-3qdh krb5kdc[21986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880712, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.34.66@KRBTEST.COM; defaulting to no policy
I20260619 14:51:52.104017 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:52.092445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:45429 (local address 127.19.34.126:44413)
0619 14:51:52.092604 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:52.092609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:52.092924 (+ 315us) server_negotiation.cc:407] Connection header received
0619 14:51:52.093901 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:52.093904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:52.093947 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:52.094021 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:52.095458 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:52.096364 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:52.097442 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:52.097845 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:52.097980 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:52.100624 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:52.100627 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:52.100630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:52.100652 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:52.102176 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:52.102625 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:52.102628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:52.102629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:52.102695 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:52.103074 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:52.103076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:52.103077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:52.103253 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:52.103327 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:52.103785 (+ 458us) server_negotiation.cc:299] Negotiation successful
0619 14:51:52.103901 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.66@KRBTEST.COM" created.
I20260619 14:51:52.104987 22224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:52.092072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44413 (local address 127.19.34.65:45429)
0619 14:51:52.092758 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:52.092824 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:52.093691 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:52.094184 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:52.094195 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:52.094670 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:52.095292 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:52.095314 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:52.096492 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:52.096495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:52.097316 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:52.097325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:52.097991 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:52.097993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:52.098120 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:52.098634 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:52.098649 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:52.100456 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:52.102290 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:52.102294 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:52.102303 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:52.102539 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:52.102820 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:52.102823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:52.102825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:52.102986 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:52.103343 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:52.103348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:52.103557 (+ 209us) client_negotiation.cc:769] Sending connection context
0619 14:51:52.103715 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 14:51:52.103998 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":144,"threads_started":1}
I20260619 14:51:52.106192 22222 heartbeater.cc:347] Connected to a master server at 127.19.34.126:44413
I20260619 14:51:52.106416 22222 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:52.106946 22222 heartbeater.cc:511] Master 127.19.34.126:44413 requested a full tablet report, sending...
I20260619 14:51:52.108367 22031 ts_manager.cc:194] Registered new tserver with Master: 2c1586c3e7fb4275a5cde6ae84cc759e (127.19.34.65:46477)
I20260619 14:51:52.109339 22031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:45429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
I20260619 14:51:52.118654 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44413
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:52.219205 22227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:52.219424 22227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:52.219465 22227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:52.223177 22227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:52.223223 22227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:52.223292 22227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:51:52.227798 22227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:52.228633 22227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:52.229483 22227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:52.238103 22235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:52.238123 22233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:52.238196 22227 server_base.cc:1034] running on GCE node
W20260619 14:51:52.238138 22232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:52.238647 22227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:52.239259 22227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:52.240450 22227 hybrid_clock.cc:648] HybridClock initialized: now 1781880712240437 us; error 32 us; skew 500 ppm
Jun 19 14:51:52 dist-test-slave-3qdh krb5kdc[21986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880712, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:52.242914 22227 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:51:52.243556 22227 webserver.cc:460] Webserver started at http://127.19.34.66:34387/ using document root <none> and password file <none>
I20260619 14:51:52.244029 22227 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:52.244103 22227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:52.244246 22227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:52.245965 22227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "5cef7f7d629845ebb6eab3fd29daa42c"
format_stamp: "Formatted at 2026-06-19 14:51:52 on dist-test-slave-3qdh"
server_key: "8d0fc97b640deb1af9b65d0b043bce45"
server_key_iv: "fa26ef01aa06b1ea5a74ca5ea5a4a9fe"
server_key_version: "encryptionkey@0"
I20260619 14:51:52.246382 22227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "5cef7f7d629845ebb6eab3fd29daa42c"
format_stamp: "Formatted at 2026-06-19 14:51:52 on dist-test-slave-3qdh"
server_key: "8d0fc97b640deb1af9b65d0b043bce45"
server_key_iv: "fa26ef01aa06b1ea5a74ca5ea5a4a9fe"
server_key_version: "encryptionkey@0"
I20260619 14:51:52.249344 22227 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 14:51:52.251338 22242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:52.252491 22227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:51:52.252568 22227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "5cef7f7d629845ebb6eab3fd29daa42c"
format_stamp: "Formatted at 2026-06-19 14:51:52 on dist-test-slave-3qdh"
server_key: "8d0fc97b640deb1af9b65d0b043bce45"
server_key_iv: "fa26ef01aa06b1ea5a74ca5ea5a4a9fe"
server_key_version: "encryptionkey@0"
I20260619 14:51:52.252633 22227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:52.263550 22227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:52.266296 22227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:52.266414 22227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:52.266796 22227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:52.267400 22227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:52.267438 22227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:52.267468 22227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:52.267486 22227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:52.275228 22227 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:35617
I20260619 14:51:52.275280 22355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:35617 every 8 connection(s)
I20260619 14:51:52.275954 22227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
Jun 19 14:51:52 dist-test-slave-3qdh krb5kdc[21986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880712, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:52.283449 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 22227
I20260619 14:51:52.283545 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:51:52.283809 19593 external_mini_cluster.cc:1428] Setting key a725e3514e27c130d39c77212e11e46f
I20260619 14:51:52.287346 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:52.277344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:42475 (local address 127.19.34.126:44413)
0619 14:51:52.277486 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:52.277490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:52.278295 (+ 805us) server_negotiation.cc:407] Connection header received
0619 14:51:52.278983 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:52.278985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:52.279026 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:52.279090 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:52.280009 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:52.280520 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:52.281413 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:52.281629 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:52.281712 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:52.283739 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:52.283742 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:52.283744 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:52.283765 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:52.285372 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:52.285890 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:52.285892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:52.285893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:52.285954 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:52.286266 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:52.286268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:52.286269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:52.286405 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:52.286467 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:52.287020 (+ 553us) server_negotiation.cc:299] Negotiation successful
0619 14:51:52.287162 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:51:52.287868 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:52.277628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44413 (local address 127.19.34.66:42475)
0619 14:51:52.278163 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:52.278198 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:52.278823 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:52.279221 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:52.279227 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:52.279506 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:52.279892 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:52.279903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:52.280609 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:52.280612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:52.281326 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:52.281334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:52.281753 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:52.281755 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:52.281870 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:52.282281 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:52.282294 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:52.283483 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:52.285471 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:52.285476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:52.285487 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:52.285799 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:52.286027 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:52.286032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:52.286036 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:52.286188 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:52.286489 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:52.286496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:52.286782 (+ 286us) client_negotiation.cc:769] Sending connection context
0619 14:51:52.286947 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 14:51:52.287151 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":88,"threads_started":1}
I20260619 14:51:52.288821 22356 heartbeater.cc:347] Connected to a master server at 127.19.34.126:44413
I20260619 14:51:52.289006 22356 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:52.289390 22356 heartbeater.cc:511] Master 127.19.34.126:44413 requested a full tablet report, sending...
I20260619 14:51:52.290227 22031 ts_manager.cc:194] Registered new tserver with Master: 5cef7f7d629845ebb6eab3fd29daa42c (127.19.34.66:35617)
I20260619 14:51:52.290730 22031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:42475
WARNING: no policy specified for kudu/127.19.34.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
I20260619 14:51:52.307313 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44413
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:37675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:52.404242 22361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:52.404508 22361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:52.404577 22361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:52.408133 22361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:52.408175 22361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:52.408282 22361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:51:52.413286 22361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:37675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:52.414104 22361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:52.414994 22361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:52.423185 22369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:52.423207 22367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:52.423204 22366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:52.423277 22361 server_base.cc:1034] running on GCE node
I20260619 14:51:52.423823 22361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:52.424412 22361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:52.425557 22361 hybrid_clock.cc:648] HybridClock initialized: now 1781880712425537 us; error 34 us; skew 500 ppm
Jun 19 14:51:52 dist-test-slave-3qdh krb5kdc[21986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880712, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:52.427793 22361 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:51:52.428620 22361 webserver.cc:460] Webserver started at http://127.19.34.67:46551/ using document root <none> and password file <none>
I20260619 14:51:52.429092 22361 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:52.429172 22361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:52.429406 22361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:52.431072 22361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "23310d328b6c4b5a949bab230c181294"
format_stamp: "Formatted at 2026-06-19 14:51:52 on dist-test-slave-3qdh"
server_key: "4ce69606543bcd6179908843793915be"
server_key_iv: "69e45f0aa24c77c4d343537bed5c1886"
server_key_version: "encryptionkey@0"
I20260619 14:51:52.431607 22361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "23310d328b6c4b5a949bab230c181294"
format_stamp: "Formatted at 2026-06-19 14:51:52 on dist-test-slave-3qdh"
server_key: "4ce69606543bcd6179908843793915be"
server_key_iv: "69e45f0aa24c77c4d343537bed5c1886"
server_key_version: "encryptionkey@0"
I20260619 14:51:52.434895 22361 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 14:51:52.437058 22376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:52.438035 22361 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 14:51:52.438112 22361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "23310d328b6c4b5a949bab230c181294"
format_stamp: "Formatted at 2026-06-19 14:51:52 on dist-test-slave-3qdh"
server_key: "4ce69606543bcd6179908843793915be"
server_key_iv: "69e45f0aa24c77c4d343537bed5c1886"
server_key_version: "encryptionkey@0"
I20260619 14:51:52.438177 22361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:52.456986 22361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:52.459364 22361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:52.459494 22361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:52.459941 22361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:52.460711 22361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:52.460754 22361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:52.460788 22361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:52.460812 22361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:52.468703 22361 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:45767
I20260619 14:51:52.468731 22489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:45767 every 8 connection(s)
I20260619 14:51:52.469448 22361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:51:52.471853 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 22361
I20260619 14:51:52.471958 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:51:52.472245 19593 external_mini_cluster.cc:1428] Setting key 66ccbc2c7e11e74b53baa26953133f94
Jun 19 14:51:52 dist-test-slave-3qdh krb5kdc[21986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880712, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:52.481534 22070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:52.471020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:53187 (local address 127.19.34.126:44413)
0619 14:51:52.471169 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:52.471173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:52.472169 (+ 996us) server_negotiation.cc:407] Connection header received
0619 14:51:52.473151 (+ 982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:52.473154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:52.473207 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:52.473308 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:52.474454 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:52.474966 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:52.475988 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:52.476179 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:52.476276 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:52.478424 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:52.478428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:52.478433 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:52.478459 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:52.479632 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:52.480111 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:52.480116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:52.480118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:52.480196 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:52.480544 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:52.480547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:52.480549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:52.480723 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:52.480828 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:52.481260 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 14:51:52.481363 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:51:52.482177 22492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:52.471377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44413 (local address 127.19.34.67:53187)
0619 14:51:52.472024 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:52.472072 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:52.472969 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:52.473456 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:52.473464 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:52.473824 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:52.474328 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:52.474340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:52.475085 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:52.475090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:52.475867 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:52.475875 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:52.476322 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:52.476327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:52.476481 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:52.476886 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:52.476899 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:52.478276 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:52.479751 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:52.479757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:52.479769 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:52.480008 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:52.480286 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:52.480290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:52.480292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:52.480442 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:52.480825 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:52.480830 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:52.481036 (+ 206us) client_negotiation.cc:769] Sending connection context
0619 14:51:52.481216 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 14:51:52.481488 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":105,"threads_started":1}
I20260619 14:51:52.483110 22490 heartbeater.cc:347] Connected to a master server at 127.19.34.126:44413
I20260619 14:51:52.483415 22490 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:52.483855 22490 heartbeater.cc:511] Master 127.19.34.126:44413 requested a full tablet report, sending...
I20260619 14:51:52.484798 22031 ts_manager.cc:194] Registered new tserver with Master: 23310d328b6c4b5a949bab230c181294 (127.19.34.67:45767)
I20260619 14:51:52.485406 22031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:53187
I20260619 14:51:52.485730 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 14:51:52.496883 22070 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:52.495756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38950 (local address 127.19.34.126:44413)
0619 14:51:52.495856 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:52.495858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:52.495990 (+ 132us) server_negotiation.cc:407] Connection header received
0619 14:51:52.496075 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:52.496077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:52.496108 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:52.496172 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:52.496802 (+ 630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38950: BlockingRecv error: recv got EOF from 127.0.0.1:38950 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:51:52.497848 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 22093
I20260619 14:51:52.502390 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 22227
I20260619 14:51:52.507385 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 22361
I20260619 14:51:52.512068 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 21999
2026-06-19T14:51:52Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1915 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:52 dist-test-slave-3qdh krb5kdc[22502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:52 dist-test-slave-3qdh krb5kdc[22502](info): set up 2 sockets
Jun 19 14:51:52 dist-test-slave-3qdh krb5kdc[22502](info): commencing operation
krb5kdc: starting...
W20260619 14:51:53.491991 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:51:53 dist-test-slave-3qdh krb5kdc[22502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:51:53.597124 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:44311
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:35973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:44311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:53.697793 22515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:53.698005 22515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:53.698041 22515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:53.701794 22515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:53.701838 22515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:53.701855 22515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:53.701870 22515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:53.701884 22515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:53.706439 22515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:44311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:44311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:53.707212 22515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:53.708040 22515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:53.715476 22521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:53.715529 22523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:53.715570 22515 server_base.cc:1034] running on GCE node
W20260619 14:51:53.715536 22520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:53.716209 22515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:53.717221 22515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:53.718369 22515 hybrid_clock.cc:648] HybridClock initialized: now 1781880713718346 us; error 35 us; skew 500 ppm
Jun 19 14:51:53 dist-test-slave-3qdh krb5kdc[22502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880713, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:53.724225 22515 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:51:53.725066 22515 webserver.cc:460] Webserver started at http://127.19.34.126:39087/ using document root <none> and password file <none>
I20260619 14:51:53.725768 22515 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:53.725862 22515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:53.726058 22515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:53.727758 22515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "49b404f9d3c242d89365cba021f6eb4a"
format_stamp: "Formatted at 2026-06-19 14:51:53 on dist-test-slave-3qdh"
server_key: "6fe107f872db9f26d25c0bcd37c0d4da"
server_key_iv: "fd4800637420a86aadb4b1959bf7475a"
server_key_version: "encryptionkey@0"
I20260619 14:51:53.728281 22515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "49b404f9d3c242d89365cba021f6eb4a"
format_stamp: "Formatted at 2026-06-19 14:51:53 on dist-test-slave-3qdh"
server_key: "6fe107f872db9f26d25c0bcd37c0d4da"
server_key_iv: "fd4800637420a86aadb4b1959bf7475a"
server_key_version: "encryptionkey@0"
I20260619 14:51:53.731473 22515 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:51:53.733568 22530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:53.734683 22515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:51:53.734818 22515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "49b404f9d3c242d89365cba021f6eb4a"
format_stamp: "Formatted at 2026-06-19 14:51:53 on dist-test-slave-3qdh"
server_key: "6fe107f872db9f26d25c0bcd37c0d4da"
server_key_iv: "fd4800637420a86aadb4b1959bf7475a"
server_key_version: "encryptionkey@0"
I20260619 14:51:53.734923 22515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:53.754753 22515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:53.757220 22515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:53.757416 22515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:53.763738 22515 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:44311
I20260619 14:51:53.763751 22582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:44311 every 8 connection(s)
I20260619 14:51:53.764567 22515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:53.766901 22583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:53.771586 22583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a: Bootstrap starting.
I20260619 14:51:53.771799 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 22515
I20260619 14:51:53.771919 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:51:53.772137 19593 external_mini_cluster.cc:1428] Setting key 45cb2dd258f1b50cf87621e71deafef0
I20260619 14:51:53.773909 22583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:53.774587 22583 log.cc:824] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:53.776412 22583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a: No bootstrap required, opened a new log
Jun 19 14:51:53 dist-test-slave-3qdh krb5kdc[22502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:53.780285 22583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b404f9d3c242d89365cba021f6eb4a" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44311 } }
I20260619 14:51:53.780539 22583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:53.780616 22583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49b404f9d3c242d89365cba021f6eb4a, State: Initialized, Role: FOLLOWER
I20260619 14:51:53.780962 22583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b404f9d3c242d89365cba021f6eb4a" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44311 } }
I20260619 14:51:53.781095 22583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:53.781148 22583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:53.781232 22583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:53.782066 22583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b404f9d3c242d89365cba021f6eb4a" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44311 } }
I20260619 14:51:53.782371 22583 leader_election.cc:302] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49b404f9d3c242d89365cba021f6eb4a; no voters:
I20260619 14:51:53.782655 22583 leader_election.cc:288] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:53.782711 22588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:53.782872 22588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a [term 1 LEADER]: Becoming Leader. State: Replica: 49b404f9d3c242d89365cba021f6eb4a, State: Running, Role: LEADER
I20260619 14:51:53.783100 22588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b404f9d3c242d89365cba021f6eb4a" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44311 } }
I20260619 14:51:53.783735 22583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:53.784818 22589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49b404f9d3c242d89365cba021f6eb4a. Latest consensus state: current_term: 1 leader_uuid: "49b404f9d3c242d89365cba021f6eb4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b404f9d3c242d89365cba021f6eb4a" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44311 } } }
I20260619 14:51:53.785121 22590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49b404f9d3c242d89365cba021f6eb4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49b404f9d3c242d89365cba021f6eb4a" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44311 } } }
I20260619 14:51:53.785221 22589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:53.785310 22590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:53.785589 22586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:53.773340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48892 (local address 127.19.34.126:44311)
0619 14:51:53.773732 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:53.773746 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:53.773778 (+ 32us) server_negotiation.cc:407] Connection header received
0619 14:51:53.774394 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:53.774410 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:53.774732 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:53.775078 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:53.775784 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:53.776742 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:53.777806 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:53.778186 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:53.778282 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:53.780778 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:53.780782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:53.780812 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:53.780840 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:53.782410 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:53.782858 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:53.782864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:53.782871 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:53.783000 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:53.783299 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:53.783302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:53.783303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:53.783641 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:53.783893 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:53.784230 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 14:51:53.784535 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":113,"threads_started":1}
I20260619 14:51:53.788353 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 14:51:53.788765 22602 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:51:53.788841 22602 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:51:53.790357 22603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:53.791205 22603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:53.796882 22603 catalog_manager.cc:1347] Generated new cluster ID: db07f585e8ca48fda4cdc7738760bf57
I20260619 14:51:53.796941 22603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:53.803838 22603 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:53.804800 22603 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:53.806607 22586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:53.801443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48906 (local address 127.19.34.126:44311)
0619 14:51:53.801538 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:53.801542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:53.801728 (+ 186us) server_negotiation.cc:407] Connection header received
0619 14:51:53.801847 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:53.801850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:53.801892 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:53.801976 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:53.802468 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:53.803166 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:53.804335 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:53.804705 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:53.804806 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:53.805740 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:53.805743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:53.805745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:53.806208 (+ 463us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:53.806238 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:53.806254 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:53.806337 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:53.806422 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 14:51:53.806494 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 14:51:53.814273 22603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49b404f9d3c242d89365cba021f6eb4a: Generated new TSK 0
I20260619 14:51:53.815043 22603 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:51:53.820032 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 22515
2026-06-19T14:51:53Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1306 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:53 dist-test-slave-3qdh krb5kdc[22616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:53 dist-test-slave-3qdh krb5kdc[22616](info): set up 2 sockets
Jun 19 14:51:53 dist-test-slave-3qdh krb5kdc[22616](info): commencing operation
krb5kdc: starting...
W20260619 14:51:54.828126 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:51:54 dist-test-slave-3qdh krb5kdc[22616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:51:54.933367 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:36547
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:43767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:36547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:55.029997 22629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:55.030236 22629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:55.030299 22629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:55.033864 22629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:55.033951 22629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:55.033990 22629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:55.034049 22629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:55.034088 22629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:55.038690 22629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:43767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:36547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:36547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:55.039448 22629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:55.040258 22629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:55.047421 22635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:55.047443 22634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:55.047462 22637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:55.047434 22629 server_base.cc:1034] running on GCE node
I20260619 14:51:55.048094 22629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:55.048988 22629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:55.050128 22629 hybrid_clock.cc:648] HybridClock initialized: now 1781880715050118 us; error 28 us; skew 500 ppm
Jun 19 14:51:55 dist-test-slave-3qdh krb5kdc[22616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880715, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:55.052440 22629 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:51:55.053186 22629 webserver.cc:460] Webserver started at http://127.19.34.126:41329/ using document root <none> and password file <none>
I20260619 14:51:55.053668 22629 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:55.053761 22629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:55.053960 22629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:55.055655 22629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "6ffb88e61a1e45c98f6175079f5e2338"
format_stamp: "Formatted at 2026-06-19 14:51:55 on dist-test-slave-3qdh"
server_key: "c8b6e00ef53031b250a7bb6c4653619d"
server_key_iv: "2eba65a8af3b15ba205b2e8e26f88b20"
server_key_version: "encryptionkey@0"
I20260619 14:51:55.056137 22629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "6ffb88e61a1e45c98f6175079f5e2338"
format_stamp: "Formatted at 2026-06-19 14:51:55 on dist-test-slave-3qdh"
server_key: "c8b6e00ef53031b250a7bb6c4653619d"
server_key_iv: "2eba65a8af3b15ba205b2e8e26f88b20"
server_key_version: "encryptionkey@0"
I20260619 14:51:55.059263 22629 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:51:55.061270 22644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:55.062278 22629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:51:55.062395 22629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "6ffb88e61a1e45c98f6175079f5e2338"
format_stamp: "Formatted at 2026-06-19 14:51:55 on dist-test-slave-3qdh"
server_key: "c8b6e00ef53031b250a7bb6c4653619d"
server_key_iv: "2eba65a8af3b15ba205b2e8e26f88b20"
server_key_version: "encryptionkey@0"
I20260619 14:51:55.062485 22629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:55.096345 22629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:55.099771 22629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:55.099937 22629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:55.105504 22629 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:36547
I20260619 14:51:55.105748 22696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:36547 every 8 connection(s)
I20260619 14:51:55.106287 22629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:55.108034 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 22629
I20260619 14:51:55.108147 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:51:55.108428 19593 external_mini_cluster.cc:1428] Setting key e29cca24df1a1b987a8d91466c794bb7
I20260619 14:51:55.108708 22697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:55.115051 22697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338: Bootstrap starting.
Jun 19 14:51:55 dist-test-slave-3qdh krb5kdc[22616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:55.117089 22697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:55.117776 22697 log.cc:824] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:55.119500 22697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338: No bootstrap required, opened a new log
I20260619 14:51:55.121218 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:55.109616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44722 (local address 127.19.34.126:36547)
0619 14:51:55.109986 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:55.110001 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:55.110036 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:51:55.110698 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:55.110717 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:55.111062 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:55.111384 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:55.112017 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:55.113434 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:55.114219 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:55.114858 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:55.115000 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:55.117301 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:55.117306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:55.117315 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:55.117336 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:55.118941 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:55.119345 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:55.119349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:55.119352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:55.119439 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:55.119696 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:55.119698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:55.119699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:55.119984 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:55.120101 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:55.120394 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 14:51:55.120629 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":96,"threads_started":1}
I20260619 14:51:55.122979 22697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffb88e61a1e45c98f6175079f5e2338" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 36547 } }
I20260619 14:51:55.123227 22697 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:55.123387 22697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ffb88e61a1e45c98f6175079f5e2338, State: Initialized, Role: FOLLOWER
I20260619 14:51:55.123965 22697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffb88e61a1e45c98f6175079f5e2338" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 36547 } }
I20260619 14:51:55.124161 22697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:55.124253 22697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:55.124366 22697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:55.125160 22697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffb88e61a1e45c98f6175079f5e2338" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 36547 } }
I20260619 14:51:55.125456 22697 leader_election.cc:302] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ffb88e61a1e45c98f6175079f5e2338; no voters:
I20260619 14:51:55.125741 22697 leader_election.cc:288] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:55.125974 22702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:55.126183 22702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338 [term 1 LEADER]: Becoming Leader. State: Replica: 6ffb88e61a1e45c98f6175079f5e2338, State: Running, Role: LEADER
I20260619 14:51:55.126432 22702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffb88e61a1e45c98f6175079f5e2338" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 36547 } }
I20260619 14:51:55.126796 22697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:55.128002 22703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ffb88e61a1e45c98f6175079f5e2338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffb88e61a1e45c98f6175079f5e2338" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 36547 } } }
I20260619 14:51:55.128077 22704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ffb88e61a1e45c98f6175079f5e2338. Latest consensus state: current_term: 1 leader_uuid: "6ffb88e61a1e45c98f6175079f5e2338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ffb88e61a1e45c98f6175079f5e2338" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 36547 } } }
I20260619 14:51:55.128160 22704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:55.128118 22703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:55.128523 22711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:55.131301 22711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:55.135685 22711 catalog_manager.cc:1347] Generated new cluster ID: 182ae5f8ce604e0f849db69ab61c9fd9
I20260619 14:51:55.135752 22711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:55.140470 22711 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:55.141546 22711 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:55.145557 22711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ffb88e61a1e45c98f6175079f5e2338: Generated new TSK 0
I20260619 14:51:55.146066 22711 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.34.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
I20260619 14:51:55.169258 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:36547
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:43767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:55.268322 22723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:55.268553 22723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:55.268622 22723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:55.272064 22723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:55.272136 22723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:55.272225 22723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:51:55.276935 22723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:43767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:36547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:55.277724 22723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:55.278543 22723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:55.286779 22731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:55.286795 22729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:55.286836 22723 server_base.cc:1034] running on GCE node
W20260619 14:51:55.286798 22728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:55.287307 22723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:55.287835 22723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:55.289016 22723 hybrid_clock.cc:648] HybridClock initialized: now 1781880715288994 us; error 38 us; skew 500 ppm
Jun 19 14:51:55 dist-test-slave-3qdh krb5kdc[22616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880715, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:55.291361 22723 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:51:55.292027 22723 webserver.cc:460] Webserver started at http://127.19.34.65:45971/ using document root <none> and password file <none>
I20260619 14:51:55.292536 22723 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:55.292608 22723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:55.292848 22723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:55.294569 22723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "61581b9da0a54af5b81a48b7fa560094"
format_stamp: "Formatted at 2026-06-19 14:51:55 on dist-test-slave-3qdh"
server_key: "45595254cc3d7aeb128cdc1f7c00ddb7"
server_key_iv: "89c60c87ccf472d09a1ce6b3a61be5f6"
server_key_version: "encryptionkey@0"
I20260619 14:51:55.295081 22723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "61581b9da0a54af5b81a48b7fa560094"
format_stamp: "Formatted at 2026-06-19 14:51:55 on dist-test-slave-3qdh"
server_key: "45595254cc3d7aeb128cdc1f7c00ddb7"
server_key_iv: "89c60c87ccf472d09a1ce6b3a61be5f6"
server_key_version: "encryptionkey@0"
I20260619 14:51:55.298177 22723 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:51:55.300189 22738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:55.301275 22723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:55.301410 22723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "61581b9da0a54af5b81a48b7fa560094"
format_stamp: "Formatted at 2026-06-19 14:51:55 on dist-test-slave-3qdh"
server_key: "45595254cc3d7aeb128cdc1f7c00ddb7"
server_key_iv: "89c60c87ccf472d09a1ce6b3a61be5f6"
server_key_version: "encryptionkey@0"
I20260619 14:51:55.301512 22723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:55.325141 22723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:55.327555 22723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:55.327725 22723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:55.328395 22723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:55.329165 22723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:55.329234 22723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:55.329294 22723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:55.329325 22723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:55.337661 22723 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:38687
I20260619 14:51:55.337671 22851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:38687 every 8 connection(s)
I20260619 14:51:55.338351 22723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:51:55.344188 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 22723
I20260619 14:51:55.344321 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:51:55.344600 19593 external_mini_cluster.cc:1428] Setting key 6f73787ee61750c138a6f635562af79d
Jun 19 14:51:55 dist-test-slave-3qdh krb5kdc[22616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880715, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:55.350651 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:55.339794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:46591 (local address 127.19.34.126:36547)
0619 14:51:55.339910 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:55.339913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:55.340555 (+ 642us) server_negotiation.cc:407] Connection header received
0619 14:51:55.341860 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:55.341863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:55.341909 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:55.341982 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:55.342930 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:55.343393 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:55.344792 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:55.345086 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:55.345184 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:55.347574 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:55.347579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:55.347586 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:55.347606 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:55.348902 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:55.349352 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:55.349356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:55.349358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:55.349430 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:55.349748 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:55.349751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:55.349752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:55.349902 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:55.349996 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:55.350381 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 14:51:55.350503 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:51:55.351389 22854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:55.340053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:36547 (local address 127.19.34.65:46591)
0619 14:51:55.340431 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:55.340463 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:55.341677 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:55.342116 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:55.342121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:55.342431 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:55.342818 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:55.342827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:55.343483 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:55.343484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:55.344624 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:55.344637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:55.345229 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:55.345232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:55.345417 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:55.345933 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:55.345953 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:55.347435 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:55.349012 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:55.349017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:55.349028 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:55.349257 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:55.349515 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:55.349519 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:55.349521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:55.349652 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:55.349995 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:55.350000 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:55.350189 (+ 189us) client_negotiation.cc:769] Sending connection context
0619 14:51:55.350345 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 14:51:55.350613 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":99,"threads_started":1}
I20260619 14:51:55.352464 22852 heartbeater.cc:347] Connected to a master server at 127.19.34.126:36547
I20260619 14:51:55.352751 22852 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.19.34.66@KRBTEST.COM; defaulting to no policy
I20260619 14:51:55.353215 22852 heartbeater.cc:511] Master 127.19.34.126:36547 requested a full tablet report, sending...
I20260619 14:51:55.354560 22660 ts_manager.cc:194] Registered new tserver with Master: 61581b9da0a54af5b81a48b7fa560094 (127.19.34.65:38687)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.66@KRBTEST.COM" created.
I20260619 14:51:55.355813 22660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:46591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
I20260619 14:51:55.368749 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:36547
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:43767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:55.467288 22857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:55.467480 22857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:55.467515 22857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:55.471486 22857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:55.471541 22857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:55.471592 22857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:51:55.476408 22857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:43767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:36547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:55.477264 22857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:55.478065 22857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:55.486385 22862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:55.486419 22863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:55.486407 22865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:55.486510 22857 server_base.cc:1034] running on GCE node
I20260619 14:51:55.487054 22857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:55.487592 22857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:55.488736 22857 hybrid_clock.cc:648] HybridClock initialized: now 1781880715488708 us; error 39 us; skew 500 ppm
Jun 19 14:51:55 dist-test-slave-3qdh krb5kdc[22616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880715, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:55.491037 22857 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:51:55.491731 22857 webserver.cc:460] Webserver started at http://127.19.34.66:36001/ using document root <none> and password file <none>
I20260619 14:51:55.492190 22857 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:55.492268 22857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:55.492479 22857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:55.494098 22857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "a47c246be67f4870baaede69d2df08c7"
format_stamp: "Formatted at 2026-06-19 14:51:55 on dist-test-slave-3qdh"
server_key: "c51e09dbfd15882d1aeee93a556f4ea1"
server_key_iv: "11dda367b6c0b84b4b8b80928d2b641b"
server_key_version: "encryptionkey@0"
I20260619 14:51:55.494561 22857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "a47c246be67f4870baaede69d2df08c7"
format_stamp: "Formatted at 2026-06-19 14:51:55 on dist-test-slave-3qdh"
server_key: "c51e09dbfd15882d1aeee93a556f4ea1"
server_key_iv: "11dda367b6c0b84b4b8b80928d2b641b"
server_key_version: "encryptionkey@0"
I20260619 14:51:55.497645 22857 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 14:51:55.499626 22872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:55.500711 22857 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:55.500790 22857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "a47c246be67f4870baaede69d2df08c7"
format_stamp: "Formatted at 2026-06-19 14:51:55 on dist-test-slave-3qdh"
server_key: "c51e09dbfd15882d1aeee93a556f4ea1"
server_key_iv: "11dda367b6c0b84b4b8b80928d2b641b"
server_key_version: "encryptionkey@0"
I20260619 14:51:55.500859 22857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:55.521456 22857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:55.523571 22857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:55.523689 22857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:55.524115 22857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:55.524860 22857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:55.524904 22857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:55.524937 22857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:55.524953 22857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:55.532737 22857 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:39967
I20260619 14:51:55.532758 22985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:39967 every 8 connection(s)
I20260619 14:51:55.533420 22857 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
Jun 19 14:51:55 dist-test-slave-3qdh krb5kdc[22616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880715, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:55.543449 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 22857
I20260619 14:51:55.543548 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:51:55.543849 19593 external_mini_cluster.cc:1428] Setting key ef3423f1d73fa20730c4c3107f45648b
I20260619 14:51:55.545194 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:55.534897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:45503 (local address 127.19.34.126:36547)
0619 14:51:55.535048 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:55.535053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:55.535750 (+ 697us) server_negotiation.cc:407] Connection header received
0619 14:51:55.536984 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:55.536986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:55.537028 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:55.537091 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:55.537963 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:55.538421 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:55.539351 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:55.539546 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:55.539612 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:55.541644 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:55.541649 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:55.541653 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:55.541683 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:55.542935 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:55.543441 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:55.543444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:55.543445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:55.543547 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:55.543938 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:55.543944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:55.543947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:55.544142 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:55.544262 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:55.544858 (+ 596us) server_negotiation.cc:299] Negotiation successful
0619 14:51:55.544993 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:51:55.545832 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:55.535246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:36547 (local address 127.19.34.66:45503)
0619 14:51:55.535611 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:55.535645 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:55.536801 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:55.537229 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:55.537234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:55.537502 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:55.537859 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:55.537869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:55.538505 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:55.538507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:55.539254 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:55.539261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:55.539647 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:55.539649 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:55.539776 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:55.540275 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:55.540288 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:55.541470 (+ 1182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:55.543041 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:55.543045 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:55.543053 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:55.543341 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:55.543648 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:55.543650 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:55.543652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:55.543808 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:55.544261 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:55.544268 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:55.544590 (+ 322us) client_negotiation.cc:769] Sending connection context
0619 14:51:55.544802 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 14:51:55.545122 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":101,"threads_started":1}
I20260619 14:51:55.546974 22986 heartbeater.cc:347] Connected to a master server at 127.19.34.126:36547
I20260619 14:51:55.547204 22986 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:55.547644 22986 heartbeater.cc:511] Master 127.19.34.126:36547 requested a full tablet report, sending...
I20260619 14:51:55.548552 22660 ts_manager.cc:194] Registered new tserver with Master: a47c246be67f4870baaede69d2df08c7 (127.19.34.66:39967)
I20260619 14:51:55.549084 22660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:45503
WARNING: no policy specified for kudu/127.19.34.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
I20260619 14:51:55.567127 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:36547
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:43767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:55.663286 22991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:55.663471 22991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:55.663506 22991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:55.667271 22991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:55.667313 22991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:55.667362 22991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:51:55.671783 22991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:43767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:36547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:55.672560 22991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:55.673357 22991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:55.681897 22996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:55.682029 22991 server_base.cc:1034] running on GCE node
W20260619 14:51:55.681957 22999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:55.681943 22997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:55.682499 22991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:55.683101 22991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:55.684266 22991 hybrid_clock.cc:648] HybridClock initialized: now 1781880715684255 us; error 34 us; skew 500 ppm
Jun 19 14:51:55 dist-test-slave-3qdh krb5kdc[22616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880715, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:55.687204 22991 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:51:55.687917 22991 webserver.cc:460] Webserver started at http://127.19.34.67:35913/ using document root <none> and password file <none>
I20260619 14:51:55.688437 22991 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:55.688496 22991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:55.688642 22991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:55.690407 22991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "824fce3ad6844c6fa4224d3a44e0039c"
format_stamp: "Formatted at 2026-06-19 14:51:55 on dist-test-slave-3qdh"
server_key: "082bb53854d6c390357e2b50b345c62b"
server_key_iv: "fe70fd4e145538866a679c24c3ba116f"
server_key_version: "encryptionkey@0"
I20260619 14:51:55.690860 22991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "824fce3ad6844c6fa4224d3a44e0039c"
format_stamp: "Formatted at 2026-06-19 14:51:55 on dist-test-slave-3qdh"
server_key: "082bb53854d6c390357e2b50b345c62b"
server_key_iv: "fe70fd4e145538866a679c24c3ba116f"
server_key_version: "encryptionkey@0"
I20260619 14:51:55.694209 22991 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:51:55.696269 23006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:55.697362 22991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:55.697444 22991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "824fce3ad6844c6fa4224d3a44e0039c"
format_stamp: "Formatted at 2026-06-19 14:51:55 on dist-test-slave-3qdh"
server_key: "082bb53854d6c390357e2b50b345c62b"
server_key_iv: "fe70fd4e145538866a679c24c3ba116f"
server_key_version: "encryptionkey@0"
I20260619 14:51:55.697515 22991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:55.710105 22991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:55.712566 22991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:55.712687 22991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:55.713138 22991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:55.713789 22991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:55.713830 22991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:55.713864 22991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:55.713899 22991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:55.721400 22991 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:43265
I20260619 14:51:55.721418 23119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:43265 every 8 connection(s)
I20260619 14:51:55.722100 22991 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
Jun 19 14:51:55 dist-test-slave-3qdh krb5kdc[22616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880715, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:55.731894 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 22991
I20260619 14:51:55.732008 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:51:55.732307 19593 external_mini_cluster.cc:1428] Setting key 22019f127efce9ba1f54017a996fec01
I20260619 14:51:55.734644 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:55.723680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:38959 (local address 127.19.34.126:36547)
0619 14:51:55.723822 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:55.723826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:55.724587 (+ 761us) server_negotiation.cc:407] Connection header received
0619 14:51:55.725910 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:55.725913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:55.725956 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:55.726026 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:55.727131 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:55.727620 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:55.728864 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:55.729106 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:55.729175 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:55.731308 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:55.731311 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:55.731313 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:55.731333 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:55.732777 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:55.733284 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:55.733289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:55.733291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:55.733361 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:55.733668 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:55.733671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:55.733672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:55.733856 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:55.733980 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:55.734448 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 14:51:55.734537 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:51:55.735303 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:55.723968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:36547 (local address 127.19.34.67:38959)
0619 14:51:55.724448 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:55.724487 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:55.725733 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:55.726182 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:55.726188 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:55.726502 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:55.727010 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:55.727030 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:55.727725 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:55.727728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:55.728763 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:55.728772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:55.729215 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:55.729218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:55.729335 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:55.729780 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:55.729794 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:55.731163 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:55.732869 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:55.732873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:55.732885 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:55.733200 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:55.733425 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:55.733428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:55.733429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:55.733580 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:55.734009 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:55.734014 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:55.734249 (+ 235us) client_negotiation.cc:769] Sending connection context
0619 14:51:55.734392 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 14:51:55.734597 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":107,"threads_started":1}
I20260619 14:51:55.736181 23120 heartbeater.cc:347] Connected to a master server at 127.19.34.126:36547
I20260619 14:51:55.736402 23120 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:55.736773 23120 heartbeater.cc:511] Master 127.19.34.126:36547 requested a full tablet report, sending...
I20260619 14:51:55.737584 22660 ts_manager.cc:194] Registered new tserver with Master: 824fce3ad6844c6fa4224d3a44e0039c (127.19.34.67:43265)
I20260619 14:51:55.738044 22660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:38959
I20260619 14:51:55.745527 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:51:55 dist-test-slave-3qdh krb5kdc[22616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880715, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 14:51:55 dist-test-slave-3qdh krb5kdc[22616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880715, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:55.767392 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:55.760027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44738 (local address 127.19.34.126:36547)
0619 14:51:55.760179 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:55.760181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:55.760268 (+ 87us) server_negotiation.cc:407] Connection header received
0619 14:51:55.760386 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:55.760388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:55.760420 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:55.760497 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:55.761105 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:55.761539 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:55.762219 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:55.762383 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:55.762445 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:55.764482 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:55.764485 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:55.764487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:55.764507 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:55.765833 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:55.766227 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:55.766232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:55.766233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:55.766303 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:55.766565 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:55.766568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:55.766569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:55.766756 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:55.766901 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:55.767136 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 14:51:55.767243 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260619 14:51:55.767729 22660 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44738
I20260619 14:51:55.768451 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 22723
I20260619 14:51:55.773501 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 22857
I20260619 14:51:55.778316 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 22991
I20260619 14:51:55.782765 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 22629
2026-06-19T14:51:55Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1963 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:55 dist-test-slave-3qdh krb5kdc[23132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:55 dist-test-slave-3qdh krb5kdc[23132](info): set up 2 sockets
Jun 19 14:51:55 dist-test-slave-3qdh krb5kdc[23132](info): commencing operation
krb5kdc: starting...
W20260619 14:51:56.744055 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:51:56 dist-test-slave-3qdh krb5kdc[23132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:51:56.852490 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:40157
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:40157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:56.947917 23145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:56.948156 23145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:56.948218 23145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:56.951772 23145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:56.951844 23145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:56.951877 23145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:56.951938 23145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:56.951969 23145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:56.956730 23145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:40157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:40157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:56.957568 23145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:56.958449 23145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:56.966070 23150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:56.966161 23153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:56.966296 23145 server_base.cc:1034] running on GCE node
W20260619 14:51:56.966070 23151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:56.966903 23145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:56.967792 23145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:56.968997 23145 hybrid_clock.cc:648] HybridClock initialized: now 1781880716968960 us; error 53 us; skew 500 ppm
Jun 19 14:51:56 dist-test-slave-3qdh krb5kdc[23132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880716, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:56.971412 23145 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:51:56.972211 23145 webserver.cc:460] Webserver started at http://127.19.34.126:41351/ using document root <none> and password file <none>
I20260619 14:51:56.972718 23145 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:56.972805 23145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:56.973012 23145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:56.974844 23145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "8e34bb48cbf448fd80e895df36d417b3"
format_stamp: "Formatted at 2026-06-19 14:51:56 on dist-test-slave-3qdh"
server_key: "9186286f0533a0dae60d32436cf101bb"
server_key_iv: "1376d887eef04c7220b3911d223fdd3a"
server_key_version: "encryptionkey@0"
I20260619 14:51:56.975383 23145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "8e34bb48cbf448fd80e895df36d417b3"
format_stamp: "Formatted at 2026-06-19 14:51:56 on dist-test-slave-3qdh"
server_key: "9186286f0533a0dae60d32436cf101bb"
server_key_iv: "1376d887eef04c7220b3911d223fdd3a"
server_key_version: "encryptionkey@0"
I20260619 14:51:56.978667 23145 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:51:56.980907 23160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:56.981969 23145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:51:56.982098 23145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "8e34bb48cbf448fd80e895df36d417b3"
format_stamp: "Formatted at 2026-06-19 14:51:56 on dist-test-slave-3qdh"
server_key: "9186286f0533a0dae60d32436cf101bb"
server_key_iv: "1376d887eef04c7220b3911d223fdd3a"
server_key_version: "encryptionkey@0"
I20260619 14:51:56.982203 23145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:56.993333 23145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:56.995886 23145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:56.996064 23145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:57.001911 23145 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:40157
I20260619 14:51:57.001933 23212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:40157 every 8 connection(s)
I20260619 14:51:57.002766 23145 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:57.005263 23213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:57.006965 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 23145
I20260619 14:51:57.007077 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:51:57.007321 19593 external_mini_cluster.cc:1428] Setting key bbac02452f198af0cc27186946db2b91
I20260619 14:51:57.010982 23213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3: Bootstrap starting.
I20260619 14:51:57.013264 23213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:57.014128 23213 log.cc:824] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:51:57 dist-test-slave-3qdh krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:57.015940 23213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3: No bootstrap required, opened a new log
I20260619 14:51:57.020100 23213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e34bb48cbf448fd80e895df36d417b3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 40157 } }
I20260619 14:51:57.020355 23213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:57.020401 23213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e34bb48cbf448fd80e895df36d417b3, State: Initialized, Role: FOLLOWER
I20260619 14:51:57.020601 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:57.008534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47382 (local address 127.19.34.126:40157)
0619 14:51:57.008900 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:57.008928 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:57.008970 (+ 42us) server_negotiation.cc:407] Connection header received
0619 14:51:57.009495 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:57.009508 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:57.009789 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:57.010076 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:57.010932 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:57.011988 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:57.013006 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:57.013451 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:57.013572 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:57.016058 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:57.016063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:57.016077 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:57.016103 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:57.017893 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:57.018264 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:57.018270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:57.018275 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:57.018398 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:57.018621 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:57.018625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:57.018629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:57.019077 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:57.019265 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:57.019472 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 14:51:57.019740 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":111,"threads_started":1}
I20260619 14:51:57.020795 23213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e34bb48cbf448fd80e895df36d417b3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 40157 } }
I20260619 14:51:57.020927 23213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:57.020985 23213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:57.021065 23213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:57.021908 23213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e34bb48cbf448fd80e895df36d417b3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 40157 } }
I20260619 14:51:57.022219 23213 leader_election.cc:302] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e34bb48cbf448fd80e895df36d417b3; no voters:
I20260619 14:51:57.022516 23213 leader_election.cc:288] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:57.022605 23218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:57.022811 23218 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3 [term 1 LEADER]: Becoming Leader. State: Replica: 8e34bb48cbf448fd80e895df36d417b3, State: Running, Role: LEADER
I20260619 14:51:57.023106 23218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e34bb48cbf448fd80e895df36d417b3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 40157 } }
I20260619 14:51:57.023550 23213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:57.024645 23220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e34bb48cbf448fd80e895df36d417b3. Latest consensus state: current_term: 1 leader_uuid: "8e34bb48cbf448fd80e895df36d417b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e34bb48cbf448fd80e895df36d417b3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 40157 } } }
I20260619 14:51:57.024778 23220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:57.025127 23227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:57.025092 23219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e34bb48cbf448fd80e895df36d417b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e34bb48cbf448fd80e895df36d417b3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 40157 } } }
I20260619 14:51:57.025254 23219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:57.027871 23227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:57.032909 23227 catalog_manager.cc:1347] Generated new cluster ID: 8ede3b4a33da44be97b396de70442c4d
I20260619 14:51:57.032976 23227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:57.040380 23227 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:57.041460 23227 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:57.056550 23227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e34bb48cbf448fd80e895df36d417b3: Generated new TSK 0
I20260619 14:51:57.057106 23227 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.34.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
I20260619 14:51:57.088847 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:40157
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:57.189086 23239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:57.189343 23239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:57.189419 23239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:57.193183 23239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:57.193248 23239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:57.193324 23239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:51:57.197944 23239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:40157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:57.198743 23239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:57.199618 23239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:57.208154 23244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:57.208166 23247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:57.208328 23239 server_base.cc:1034] running on GCE node
W20260619 14:51:57.208166 23245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:57.208822 23239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:57.209369 23239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:57.210510 23239 hybrid_clock.cc:648] HybridClock initialized: now 1781880717210497 us; error 26 us; skew 500 ppm
Jun 19 14:51:57 dist-test-slave-3qdh krb5kdc[23132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880717, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:57.212980 23239 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:51:57.213678 23239 webserver.cc:460] Webserver started at http://127.19.34.65:43859/ using document root <none> and password file <none>
I20260619 14:51:57.214136 23239 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:57.214197 23239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:57.214339 23239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:57.216034 23239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "3e53aa2f995d478e9e4323e2719d260a"
format_stamp: "Formatted at 2026-06-19 14:51:57 on dist-test-slave-3qdh"
server_key: "53e1d9aa5489de0ffe94a36fe6cb13f1"
server_key_iv: "05f73f9a1135a176033ba102b7566980"
server_key_version: "encryptionkey@0"
I20260619 14:51:57.216548 23239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "3e53aa2f995d478e9e4323e2719d260a"
format_stamp: "Formatted at 2026-06-19 14:51:57 on dist-test-slave-3qdh"
server_key: "53e1d9aa5489de0ffe94a36fe6cb13f1"
server_key_iv: "05f73f9a1135a176033ba102b7566980"
server_key_version: "encryptionkey@0"
I20260619 14:51:57.219731 23239 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:51:57.221851 23254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:57.222892 23239 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260619 14:51:57.223006 23239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "3e53aa2f995d478e9e4323e2719d260a"
format_stamp: "Formatted at 2026-06-19 14:51:57 on dist-test-slave-3qdh"
server_key: "53e1d9aa5489de0ffe94a36fe6cb13f1"
server_key_iv: "05f73f9a1135a176033ba102b7566980"
server_key_version: "encryptionkey@0"
I20260619 14:51:57.223115 23239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:57.232034 23239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:57.234493 23239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:57.234668 23239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:57.235157 23239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:57.235841 23239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:57.235903 23239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:57.235961 23239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:57.235992 23239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:57.244518 23239 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:42075
I20260619 14:51:57.244588 23367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:42075 every 8 connection(s)
I20260619 14:51:57.245374 23239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
Jun 19 14:51:57 dist-test-slave-3qdh krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880717, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:57.253768 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 23239
I20260619 14:51:57.253893 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:51:57.254158 19593 external_mini_cluster.cc:1428] Setting key 79cbf3807ea3f425d4be8945cce139db
I20260619 14:51:57.257566 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:57.247180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:39149 (local address 127.19.34.126:40157)
0619 14:51:57.248132 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:57.248135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:57.248149 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:51:57.249232 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:57.249234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:57.249274 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:57.249337 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:57.250303 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:57.250790 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:57.251755 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:57.251945 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:57.252008 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:57.254198 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:57.254202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:57.254204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:57.254226 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:57.255685 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:57.256168 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:57.256172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:57.256174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:57.256257 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:57.256611 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:57.256615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:57.256616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:57.256781 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:57.256880 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:57.257297 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 14:51:57.257409 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":872}
I20260619 14:51:57.258168 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:57.247476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:40157 (local address 127.19.34.65:39149)
0619 14:51:57.247896 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:57.247932 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:57.249055 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:57.249476 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:57.249482 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:57.249789 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:57.250186 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:57.250198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:57.250886 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:57.250888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:57.251660 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:57.251669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:57.252054 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:57.252056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:57.252183 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:57.252660 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:57.252674 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:57.253998 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:57.255792 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:57.255798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:57.255809 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:57.256069 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:57.256380 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:57.256383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:57.256385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:57.256512 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:57.256888 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:57.256895 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:57.257089 (+ 194us) client_negotiation.cc:769] Sending connection context
0619 14:51:57.257256 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 14:51:57.257503 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":84,"threads_started":1}
I20260619 14:51:57.259245 23368 heartbeater.cc:347] Connected to a master server at 127.19.34.126:40157
I20260619 14:51:57.259440 23368 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:57.259927 23368 heartbeater.cc:511] Master 127.19.34.126:40157 requested a full tablet report, sending...
I20260619 14:51:57.261526 23177 ts_manager.cc:194] Registered new tserver with Master: 3e53aa2f995d478e9e4323e2719d260a (127.19.34.65:42075)
WARNING: no policy specified for kudu/127.19.34.66@KRBTEST.COM; defaulting to no policy
I20260619 14:51:57.262903 23177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:39149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
I20260619 14:51:57.278363 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:40157
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:57.375507 23373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:57.375692 23373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:57.375725 23373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:57.379278 23373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:57.379323 23373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:57.379374 23373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:51:57.383697 23373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:40157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:57.384542 23373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:57.385416 23373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:57.393298 23381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:57.393416 23373 server_base.cc:1034] running on GCE node
W20260619 14:51:57.393342 23379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:57.393280 23378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:57.393822 23373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:57.394378 23373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:57.395526 23373 hybrid_clock.cc:648] HybridClock initialized: now 1781880717395507 us; error 30 us; skew 500 ppm
Jun 19 14:51:57 dist-test-slave-3qdh krb5kdc[23132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880717, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:57.397984 23373 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:51:57.398725 23373 webserver.cc:460] Webserver started at http://127.19.34.66:42973/ using document root <none> and password file <none>
I20260619 14:51:57.399209 23373 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:57.399264 23373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:57.399471 23373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:57.401232 23373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "cbe7015060e447f79ae8595b4db3c04e"
format_stamp: "Formatted at 2026-06-19 14:51:57 on dist-test-slave-3qdh"
server_key: "0d77c03d15917ed17e02a3d244e527cb"
server_key_iv: "2c59fd82ac656c14c95e8e6c689a1d05"
server_key_version: "encryptionkey@0"
I20260619 14:51:57.401729 23373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "cbe7015060e447f79ae8595b4db3c04e"
format_stamp: "Formatted at 2026-06-19 14:51:57 on dist-test-slave-3qdh"
server_key: "0d77c03d15917ed17e02a3d244e527cb"
server_key_iv: "2c59fd82ac656c14c95e8e6c689a1d05"
server_key_version: "encryptionkey@0"
I20260619 14:51:57.404874 23373 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:51:57.406951 23388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:57.408046 23373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:57.408160 23373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "cbe7015060e447f79ae8595b4db3c04e"
format_stamp: "Formatted at 2026-06-19 14:51:57 on dist-test-slave-3qdh"
server_key: "0d77c03d15917ed17e02a3d244e527cb"
server_key_iv: "2c59fd82ac656c14c95e8e6c689a1d05"
server_key_version: "encryptionkey@0"
I20260619 14:51:57.408257 23373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:57.424049 23373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:57.426888 23373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:57.427042 23373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:57.427482 23373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:57.428141 23373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:57.428227 23373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:57.428289 23373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:57.428356 23373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:57.436757 23373 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:36343
I20260619 14:51:57.436772 23501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:36343 every 8 connection(s)
I20260619 14:51:57.437511 23373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:51:57.443179 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 23373
I20260619 14:51:57.443302 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:51:57.443588 19593 external_mini_cluster.cc:1428] Setting key 275dea173fbb54fb542889f86ecf0de1
Jun 19 14:51:57 dist-test-slave-3qdh krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880717, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:57.451092 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:57.439046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:34059 (local address 127.19.34.126:40157)
0619 14:51:57.439210 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:57.439214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:57.439940 (+ 726us) server_negotiation.cc:407] Connection header received
0619 14:51:57.441186 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:57.441189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:57.441238 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:57.441308 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:57.442301 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:57.442832 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:57.444349 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:57.444673 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:57.444774 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:57.447377 (+ 2603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:57.447383 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:57.447388 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:57.447419 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:57.449144 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:57.449611 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:57.449616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:57.449619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:57.449729 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:57.450016 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:57.450022 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:57.450026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:57.450258 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:57.450366 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:57.450745 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 14:51:57.450869 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":58}
I20260619 14:51:57.451519 23504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:57.439420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:40157 (local address 127.19.34.66:34059)
0619 14:51:57.439791 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:57.439828 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:57.440990 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:57.441448 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:57.441454 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:57.441754 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:57.442176 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:57.442200 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:57.442928 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:57.442930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:57.444209 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:57.444218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:57.444838 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:57.444842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:57.445019 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:57.445556 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:57.445590 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:57.447217 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:57.449280 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:57.449287 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:57.449300 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:57.449521 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:57.449819 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:57.449821 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:57.449822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:57.449929 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:57.450358 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:57.450362 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:57.450531 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 14:51:57.450657 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 14:51:57.450852 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":94,"threads_started":1}
I20260619 14:51:57.452663 23502 heartbeater.cc:347] Connected to a master server at 127.19.34.126:40157
I20260619 14:51:57.452860 23502 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:57.453245 23502 heartbeater.cc:511] Master 127.19.34.126:40157 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.34.67@KRBTEST.COM; defaulting to no policy
I20260619 14:51:57.454190 23177 ts_manager.cc:194] Registered new tserver with Master: cbe7015060e447f79ae8595b4db3c04e (127.19.34.66:36343)
I20260619 14:51:57.454818 23177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:34059
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
I20260619 14:51:57.468698 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:40157
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:57.567703 23507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:57.567935 23507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:57.567999 23507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:57.571704 23507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:57.571779 23507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:57.571858 23507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:51:57.576591 23507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:40157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:57.577435 23507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:57.578269 23507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:57.586377 23512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:57.586447 23507 server_base.cc:1034] running on GCE node
W20260619 14:51:57.586472 23515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:57.586377 23513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:57.586979 23507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:57.587473 23507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:57.588647 23507 hybrid_clock.cc:648] HybridClock initialized: now 1781880717588608 us; error 47 us; skew 500 ppm
Jun 19 14:51:57 dist-test-slave-3qdh krb5kdc[23132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880717, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:57.591161 23507 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:51:57.591889 23507 webserver.cc:460] Webserver started at http://127.19.34.67:44925/ using document root <none> and password file <none>
I20260619 14:51:57.592376 23507 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:57.592437 23507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:57.592576 23507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:57.594326 23507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "2ec98efefa7c4d81a2930e8aa3db73c5"
format_stamp: "Formatted at 2026-06-19 14:51:57 on dist-test-slave-3qdh"
server_key: "7d18559626c0ffca7c896d157f0224c4"
server_key_iv: "b963dab6aabf776036ffcd7001dec7c5"
server_key_version: "encryptionkey@0"
I20260619 14:51:57.594781 23507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "2ec98efefa7c4d81a2930e8aa3db73c5"
format_stamp: "Formatted at 2026-06-19 14:51:57 on dist-test-slave-3qdh"
server_key: "7d18559626c0ffca7c896d157f0224c4"
server_key_iv: "b963dab6aabf776036ffcd7001dec7c5"
server_key_version: "encryptionkey@0"
I20260619 14:51:57.597843 23507 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:51:57.599923 23522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:57.601130 23507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:57.601226 23507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "2ec98efefa7c4d81a2930e8aa3db73c5"
format_stamp: "Formatted at 2026-06-19 14:51:57 on dist-test-slave-3qdh"
server_key: "7d18559626c0ffca7c896d157f0224c4"
server_key_iv: "b963dab6aabf776036ffcd7001dec7c5"
server_key_version: "encryptionkey@0"
I20260619 14:51:57.601306 23507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:57.618155 23507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:57.620522 23507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:57.620636 23507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:57.621017 23507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:57.621620 23507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:57.621659 23507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:57.621690 23507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:57.621703 23507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:57.629855 23507 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:40881
I20260619 14:51:57.629890 23635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:40881 every 8 connection(s)
I20260619 14:51:57.630566 23507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:51:57.633267 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 23507
I20260619 14:51:57.633345 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:51:57.633579 19593 external_mini_cluster.cc:1428] Setting key 57327fbc0cead5e056a3473f55280eee
Jun 19 14:51:57 dist-test-slave-3qdh krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880717, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:57.642170 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:57.631938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:34617 (local address 127.19.34.126:40157)
0619 14:51:57.632056 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:57.632059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:57.632718 (+ 659us) server_negotiation.cc:407] Connection header received
0619 14:51:57.634037 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:57.634041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:57.634110 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:57.634209 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:57.635553 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:57.636038 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:57.637015 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:57.637198 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:57.637266 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:57.639352 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:57.639354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:57.639356 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:57.639377 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:57.640628 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:57.641059 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:57.641061 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:57.641062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:57.641139 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:57.641411 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:57.641414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:57.641415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:57.641561 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:57.641632 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:57.641995 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 14:51:57.642074 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:51:57.642822 23638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:57.632236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:40157 (local address 127.19.34.67:34617)
0619 14:51:57.632593 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:57.632625 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:57.633806 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:57.634376 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:57.634386 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:57.634838 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:57.635430 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:57.635448 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:57.636131 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:57.636133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:57.636927 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:57.636935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:57.637301 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:57.637303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:57.637411 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:57.637973 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:57.637986 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:57.639216 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:57.640739 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:57.640743 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:57.640751 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:57.640978 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:57.641216 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:57.641218 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:57.641219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:57.641320 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:57.641641 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:57.641644 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:57.641819 (+ 175us) client_negotiation.cc:769] Sending connection context
0619 14:51:57.641937 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 14:51:57.642116 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":82,"threads_started":1}
I20260619 14:51:57.643766 23636 heartbeater.cc:347] Connected to a master server at 127.19.34.126:40157
I20260619 14:51:57.643955 23636 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:57.644404 23636 heartbeater.cc:511] Master 127.19.34.126:40157 requested a full tablet report, sending...
I20260619 14:51:57.645326 23177 ts_manager.cc:194] Registered new tserver with Master: 2ec98efefa7c4d81a2930e8aa3db73c5 (127.19.34.67:40881)
I20260619 14:51:57.645879 23177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:34617
I20260619 14:51:57.647179 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:51:57 dist-test-slave-3qdh krb5kdc[23132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:51:57 dist-test-slave-3qdh krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.65@KRBTEST.COM
I20260619 14:51:57.669543 23643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:57.661617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34060 (local address 127.19.34.65:42075)
0619 14:51:57.661906 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:57.661913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:57.661943 (+ 30us) server_negotiation.cc:407] Connection header received
0619 14:51:57.661981 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:57.661984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:57.662111 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:57.662214 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:57.662819 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:57.663530 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:57.664449 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:57.664781 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:57.664863 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:57.666856 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:57.666863 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:57.666872 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:57.666893 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:57.668219 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:57.668584 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:57.668588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:57.668590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:57.668663 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:57.668860 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:57.668862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:57.668863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:57.669044 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:57.669197 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:57.669317 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 14:51:57.669417 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":105,"threads_started":1}
Jun 19 14:51:57 dist-test-slave-3qdh krb5kdc[23132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:57.678727 23216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:57.671822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47386 (local address 127.19.34.126:40157)
0619 14:51:57.671930 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:57.671933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:57.672007 (+ 74us) server_negotiation.cc:407] Connection header received
0619 14:51:57.672073 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:57.672074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:57.672106 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:57.672181 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:57.672796 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:57.673261 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:57.673935 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:57.674154 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:57.674224 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:57.676170 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:57.676172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:57.676174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:57.676193 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:57.677458 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:57.677804 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:57.677807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:57.677808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:57.677880 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:57.678111 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:57.678113 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:57.678114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:57.678302 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:57.678395 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:57.678558 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 14:51:57.678632 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 14:51:57.679018 23177 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47386
I20260619 14:51:57.679553 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 23239
I20260619 14:51:57.684567 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 23373
I20260619 14:51:57.689302 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 23507
I20260619 14:51:57.693830 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 23145
2026-06-19T14:51:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1911 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:57 dist-test-slave-3qdh krb5kdc[23648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:57 dist-test-slave-3qdh krb5kdc[23648](info): set up 2 sockets
Jun 19 14:51:57 dist-test-slave-3qdh krb5kdc[23648](info): commencing operation
krb5kdc: starting...
W20260619 14:51:58.668037 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:51:58 dist-test-slave-3qdh krb5kdc[23648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:51:58.772946 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:33521
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:33521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:58.870879 23661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:58.871068 23661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:58.871100 23661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:58.874501 23661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:58.874547 23661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:58.874567 23661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:58.874583 23661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:58.874598 23661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:58.878912 23661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:33521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:33521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:58.879662 23661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:58.880452 23661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:51:58.887555 23661 server_base.cc:1034] running on GCE node
W20260619 14:51:58.887524 23666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:58.887530 23669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:58.887514 23667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:58.888213 23661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:58.889148 23661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:58.890314 23661 hybrid_clock.cc:648] HybridClock initialized: now 1781880718890283 us; error 43 us; skew 500 ppm
Jun 19 14:51:58 dist-test-slave-3qdh krb5kdc[23648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880718, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:58.896220 23661 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:51:58.896790 23661 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:58.896842 23661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:58.896972 23661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:58.898510 23661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "d923bb0b54d249f488bf5ce4b381e300"
format_stamp: "Formatted at 2026-06-19 14:51:58 on dist-test-slave-3qdh"
server_key: "ee7a6470a0d9449384298008ebe23081"
server_key_iv: "626844504a9f2a28ad757e856e5ba06f"
server_key_version: "encryptionkey@0"
I20260619 14:51:58.898923 23661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "d923bb0b54d249f488bf5ce4b381e300"
format_stamp: "Formatted at 2026-06-19 14:51:58 on dist-test-slave-3qdh"
server_key: "ee7a6470a0d9449384298008ebe23081"
server_key_iv: "626844504a9f2a28ad757e856e5ba06f"
server_key_version: "encryptionkey@0"
I20260619 14:51:58.901827 23661 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:51:58.903782 23675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:58.904896 23661 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:58.905017 23661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "d923bb0b54d249f488bf5ce4b381e300"
format_stamp: "Formatted at 2026-06-19 14:51:58 on dist-test-slave-3qdh"
server_key: "ee7a6470a0d9449384298008ebe23081"
server_key_iv: "626844504a9f2a28ad757e856e5ba06f"
server_key_version: "encryptionkey@0"
I20260619 14:51:58.905109 23661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:58.920106 23661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:58.922829 23661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:58.923002 23661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:58.928820 23661 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:33521
I20260619 14:51:58.929044 23727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:33521 every 8 connection(s)
I20260619 14:51:58.929368 23661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:58.931520 23728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:58.936241 23728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300: Bootstrap starting.
I20260619 14:51:58.937866 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 23661
I20260619 14:51:58.937963 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:51:58.938198 19593 external_mini_cluster.cc:1428] Setting key c4504e5a8af36eb9ae03aa22c1c81aab
I20260619 14:51:58.938468 23728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:58.939221 23728 log.cc:824] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:58.940987 23728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300: No bootstrap required, opened a new log
I20260619 14:51:58.944510 23728 raft_consensus.cc:348] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d923bb0b54d249f488bf5ce4b381e300" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33521 } }
I20260619 14:51:58.944777 23728 raft_consensus.cc:374] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:58.944852 23728 raft_consensus.cc:729] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d923bb0b54d249f488bf5ce4b381e300, State: Initialized, Role: FOLLOWER
I20260619 14:51:58.945330 23728 consensus_queue.cc:260] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d923bb0b54d249f488bf5ce4b381e300" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33521 } }
I20260619 14:51:58.945454 23728 raft_consensus.cc:388] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:58.945525 23728 raft_consensus.cc:482] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:58.945624 23728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 14:51:58 dist-test-slave-3qdh krb5kdc[23648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:58.946425 23728 raft_consensus.cc:504] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d923bb0b54d249f488bf5ce4b381e300" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33521 } }
I20260619 14:51:58.946743 23728 leader_election.cc:302] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d923bb0b54d249f488bf5ce4b381e300; no voters:
I20260619 14:51:58.947049 23728 leader_election.cc:288] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:58.947134 23733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:58.947325 23733 raft_consensus.cc:686] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300 [term 1 LEADER]: Becoming Leader. State: Replica: d923bb0b54d249f488bf5ce4b381e300, State: Running, Role: LEADER
I20260619 14:51:58.947556 23733 consensus_queue.cc:237] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d923bb0b54d249f488bf5ce4b381e300" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33521 } }
I20260619 14:51:58.948180 23728 sys_catalog.cc:564] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:58.948874 23734 sys_catalog.cc:455] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d923bb0b54d249f488bf5ce4b381e300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d923bb0b54d249f488bf5ce4b381e300" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33521 } } }
I20260619 14:51:58.949007 23734 sys_catalog.cc:458] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:58.949323 23735 sys_catalog.cc:455] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d923bb0b54d249f488bf5ce4b381e300. Latest consensus state: current_term: 1 leader_uuid: "d923bb0b54d249f488bf5ce4b381e300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d923bb0b54d249f488bf5ce4b381e300" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33521 } } }
I20260619 14:51:58.949415 23735 sys_catalog.cc:458] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:58.951224 23731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:58.939119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44042 (local address 127.19.34.126:33521)
0619 14:51:58.939431 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:58.939448 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:58.939476 (+ 28us) server_negotiation.cc:407] Connection header received
0619 14:51:58.940113 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:58.940128 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:58.940441 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:58.940662 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:58.941413 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:58.942206 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:58.943213 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:58.943581 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:58.943682 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:58.946313 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:58.946320 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:58.946335 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:58.946366 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:58.948154 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:58.948648 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:58.948654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:58.948659 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:58.948783 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:58.949082 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:58.949086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:58.949089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:58.949589 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:58.949762 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:58.950085 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 14:51:58.950388 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":98,"threads_started":1}
W20260619 14:51:58.952723 23748 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:51:58.952805 23748 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:51:58.952883 23742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:58.953644 23742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:58.957683 23742 catalog_manager.cc:1347] Generated new cluster ID: 68f8ec9029e146bb9985e494ffc83950
I20260619 14:51:58.957736 23742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:58.965935 23742 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:58.966776 23742 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:58.971159 23742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300: Generated new TSK 0
I20260619 14:51:58.971673 23742 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.34.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
I20260619 14:51:59.000519 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:33521
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:59.120144 23754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:59.120412 23754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:59.120486 23754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:59.123945 23754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:59.124027 23754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:59.124115 23754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:51:59.128840 23754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:33521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23754
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:59.129655 23754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:59.130476 23754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:59.138453 23760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:59.138449 23759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:59.138538 23762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:59.138662 23754 server_base.cc:1034] running on GCE node
I20260619 14:51:59.139076 23754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:59.139648 23754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:59.140790 23754 hybrid_clock.cc:648] HybridClock initialized: now 1781880719140775 us; error 25 us; skew 500 ppm
Jun 19 14:51:59 dist-test-slave-3qdh krb5kdc[23648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880719, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:59.143272 23754 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:51:59.143899 23754 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:59.143996 23754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:59.144198 23754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:59.146013 23754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "4d7708b9b74846d194d528edf9873229"
format_stamp: "Formatted at 2026-06-19 14:51:59 on dist-test-slave-3qdh"
server_key: "c2a9b73ea08591fd9988967e3cc9f864"
server_key_iv: "c7c58dbe8c1dfb55c4b16067663d2869"
server_key_version: "encryptionkey@0"
I20260619 14:51:59.146488 23754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "4d7708b9b74846d194d528edf9873229"
format_stamp: "Formatted at 2026-06-19 14:51:59 on dist-test-slave-3qdh"
server_key: "c2a9b73ea08591fd9988967e3cc9f864"
server_key_iv: "c7c58dbe8c1dfb55c4b16067663d2869"
server_key_version: "encryptionkey@0"
I20260619 14:51:59.149920 23754 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:51:59.152089 23768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:59.153086 23754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:51:59.153213 23754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "4d7708b9b74846d194d528edf9873229"
format_stamp: "Formatted at 2026-06-19 14:51:59 on dist-test-slave-3qdh"
server_key: "c2a9b73ea08591fd9988967e3cc9f864"
server_key_iv: "c7c58dbe8c1dfb55c4b16067663d2869"
server_key_version: "encryptionkey@0"
I20260619 14:51:59.153301 23754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:59.163264 23754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:59.165825 23754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:59.166003 23754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:59.166462 23754 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:59.167138 23754 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:59.167215 23754 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:59.167276 23754 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:59.167320 23754 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:59.175246 23754 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:32971
I20260619 14:51:59.175266 23881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:32971 every 8 connection(s)
I20260619 14:51:59.175721 23754 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
Jun 19 14:51:59 dist-test-slave-3qdh krb5kdc[23648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880719, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:59.185701 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 23754
I20260619 14:51:59.185786 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:51:59.186028 19593 external_mini_cluster.cc:1428] Setting key e8839d148aafbbd7b3a2bc5416e3d24e
I20260619 14:51:59.186939 23731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:59.177148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:41171 (local address 127.19.34.126:33521)
0619 14:51:59.177288 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:59.177292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:59.177973 (+ 681us) server_negotiation.cc:407] Connection header received
0619 14:51:59.179051 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:59.179053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:59.179093 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:59.179154 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:59.180046 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.180536 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.181456 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.181645 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.181710 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:59.183524 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:59.183526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:59.183528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:59.183548 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:59.184916 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:59.185412 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:59.185414 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:59.185415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:59.185522 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:59.185907 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:59.185911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:59.185913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:59.186086 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:59.186213 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:59.186680 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 14:51:59.186800 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:51:59.187569 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:59.177437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:33521 (local address 127.19.34.65:41171)
0619 14:51:59.177839 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:59.177876 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:59.178878 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:59.179288 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:59.179293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:59.179553 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:59.179938 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:59.179948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.180630 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.180632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:59.181360 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:59.181367 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.181755 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.181758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:59.181870 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:59.182247 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:59.182260 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:59.183396 (+ 1136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:59.185022 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:59.185026 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:59.185036 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:59.185312 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:59.185611 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:59.185613 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:59.185614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:59.185736 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:59.186230 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:59.186236 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:59.186419 (+ 183us) client_negotiation.cc:769] Sending connection context
0619 14:51:59.186614 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 14:51:59.186828 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":92,"threads_started":1}
I20260619 14:51:59.188799 23882 heartbeater.cc:347] Connected to a master server at 127.19.34.126:33521
I20260619 14:51:59.189011 23882 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:59.189365 23882 heartbeater.cc:511] Master 127.19.34.126:33521 requested a full tablet report, sending...
I20260619 14:51:59.190616 23692 ts_manager.cc:194] Registered new tserver with Master: 4d7708b9b74846d194d528edf9873229 (127.19.34.65:32971)
I20260619 14:51:59.191733 23692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:41171
WARNING: no policy specified for kudu/127.19.34.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
I20260619 14:51:59.208775 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:33521
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:59.306530 23887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:59.306721 23887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:59.306758 23887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:59.310454 23887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:59.310504 23887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:59.310554 23887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:51:59.315219 23887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:33521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:59.315989 23887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:59.316771 23887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:59.324496 23892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:59.324566 23895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:59.324719 23893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:59.324759 23887 server_base.cc:1034] running on GCE node
I20260619 14:51:59.325186 23887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:59.325750 23887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:59.326932 23887 hybrid_clock.cc:648] HybridClock initialized: now 1781880719326919 us; error 32 us; skew 500 ppm
Jun 19 14:51:59 dist-test-slave-3qdh krb5kdc[23648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880719, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:59.329515 23887 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:51:59.330156 23887 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:59.330241 23887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:59.330446 23887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:59.332113 23887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "b94a91900e644c249e285df11925d757"
format_stamp: "Formatted at 2026-06-19 14:51:59 on dist-test-slave-3qdh"
server_key: "99fa87bce148c16d61fd375ed71677c5"
server_key_iv: "8674219da88dcbf95ec0641a4bf30a82"
server_key_version: "encryptionkey@0"
I20260619 14:51:59.332746 23887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "b94a91900e644c249e285df11925d757"
format_stamp: "Formatted at 2026-06-19 14:51:59 on dist-test-slave-3qdh"
server_key: "99fa87bce148c16d61fd375ed71677c5"
server_key_iv: "8674219da88dcbf95ec0641a4bf30a82"
server_key_version: "encryptionkey@0"
I20260619 14:51:59.336108 23887 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:51:59.338372 23901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:59.339344 23887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:59.339479 23887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "b94a91900e644c249e285df11925d757"
format_stamp: "Formatted at 2026-06-19 14:51:59 on dist-test-slave-3qdh"
server_key: "99fa87bce148c16d61fd375ed71677c5"
server_key_iv: "8674219da88dcbf95ec0641a4bf30a82"
server_key_version: "encryptionkey@0"
I20260619 14:51:59.339576 23887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:59.365025 23887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:59.368876 23887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:59.369086 23887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:59.369686 23887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:59.370559 23887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:59.370625 23887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:59.370682 23887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:59.370711 23887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:59.382303 23887 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:38767
I20260619 14:51:59.382359 24014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:38767 every 8 connection(s)
I20260619 14:51:59.383018 23887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:51:59.383579 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 23887
I20260619 14:51:59.383702 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:51:59.383970 19593 external_mini_cluster.cc:1428] Setting key b3d0ad96cb62eb474bd71d74fd3c5def
WARNING: no policy specified for kudu/127.19.34.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.67@KRBTEST.COM" created.
Jun 19 14:51:59 dist-test-slave-3qdh krb5kdc[23648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880719, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:59.402266 23731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:59.384755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:48867 (local address 127.19.34.126:33521)
0619 14:51:59.386309 (+ 1554us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:59.386314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:59.386328 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:51:59.387863 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:59.387870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:59.387942 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:59.388023 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:59.389481 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.390238 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.391926 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.392383 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.392487 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:59.397882 (+ 5395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:59.397888 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:59.397893 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:59.397925 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:59.399661 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:59.400379 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:59.400385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:59.400388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:59.400494 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:59.400971 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:59.400977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:59.400978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:59.401260 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:59.401354 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:59.401964 (+ 610us) server_negotiation.cc:299] Negotiation successful
0619 14:51:59.402076 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1490}
I20260619 14:51:59.402868 24017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:59.385210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:33521 (local address 127.19.34.66:48867)
0619 14:51:59.385888 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:59.385972 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:59.387628 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:59.388191 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:59.388200 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:59.388718 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:59.389340 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:59.389360 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.390363 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.390369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:59.391703 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:59.391733 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.392732 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.392738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:59.393015 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:59.393640 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:59.393663 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:59.397689 (+ 4026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:59.399802 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:59.399810 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:59.399823 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:59.400212 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:59.400647 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:59.400652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:59.400654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:59.400838 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:59.401428 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:59.401435 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:59.401726 (+ 291us) client_negotiation.cc:769] Sending connection context
0619 14:51:59.401907 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 14:51:59.402184 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":147,"threads_started":1}
I20260619 14:51:59.403990 24015 heartbeater.cc:347] Connected to a master server at 127.19.34.126:33521
I20260619 14:51:59.404246 24015 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:59.404714 24015 heartbeater.cc:511] Master 127.19.34.126:33521 requested a full tablet report, sending...
I20260619 14:51:59.405685 23692 ts_manager.cc:194] Registered new tserver with Master: b94a91900e644c249e285df11925d757 (127.19.34.66:38767)
I20260619 14:51:59.406277 23692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:48867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
I20260619 14:51:59.411294 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:33521
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:39369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:59.509171 24020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:59.509388 24020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:59.509442 24020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:59.513089 24020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:59.513127 24020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:59.513176 24020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:51:59.517879 24020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:39369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:33521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:59.518594 24020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:59.519348 24020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:59.527204 24025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:59.527211 24026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:59.527314 24020 server_base.cc:1034] running on GCE node
W20260619 14:51:59.527256 24028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:59.527714 24020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:59.528261 24020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:59.529448 24020 hybrid_clock.cc:648] HybridClock initialized: now 1781880719529431 us; error 25 us; skew 500 ppm
Jun 19 14:51:59 dist-test-slave-3qdh krb5kdc[23648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880719, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:59.531740 24020 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:51:59.532404 24020 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:59.532466 24020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:59.532610 24020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:59.534299 24020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "659fa6281f474d358859e7cfec6bf2f8"
format_stamp: "Formatted at 2026-06-19 14:51:59 on dist-test-slave-3qdh"
server_key: "ed560e277fe9580ebc93732553bce0c0"
server_key_iv: "589905fae118573bddd6968839caa076"
server_key_version: "encryptionkey@0"
I20260619 14:51:59.534749 24020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "659fa6281f474d358859e7cfec6bf2f8"
format_stamp: "Formatted at 2026-06-19 14:51:59 on dist-test-slave-3qdh"
server_key: "ed560e277fe9580ebc93732553bce0c0"
server_key_iv: "589905fae118573bddd6968839caa076"
server_key_version: "encryptionkey@0"
I20260619 14:51:59.537881 24020 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260619 14:51:59.539775 24034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:59.540831 24020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:51:59.540942 24020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "659fa6281f474d358859e7cfec6bf2f8"
format_stamp: "Formatted at 2026-06-19 14:51:59 on dist-test-slave-3qdh"
server_key: "ed560e277fe9580ebc93732553bce0c0"
server_key_iv: "589905fae118573bddd6968839caa076"
server_key_version: "encryptionkey@0"
I20260619 14:51:59.541025 24020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:59.573601 24020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:59.575984 24020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:59.576157 24020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:59.576658 24020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:51:59.577342 24020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:51:59.577402 24020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:59.577459 24020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:51:59.577500 24020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:59.585474 24020 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:35211
I20260619 14:51:59.585496 24147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:35211 every 8 connection(s)
I20260619 14:51:59.585911 24020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:51:59.586442 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 24020
I20260619 14:51:59.586565 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:51:59.586822 19593 external_mini_cluster.cc:1428] Setting key c77c240d55c3722496b9590f7996caea
Jun 19 14:51:59 dist-test-slave-3qdh krb5kdc[23648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880719, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:51:59.597853 23731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:59.587498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:51525 (local address 127.19.34.126:33521)
0619 14:51:59.587642 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:59.587646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:59.588339 (+ 693us) server_negotiation.cc:407] Connection header received
0619 14:51:59.589538 (+ 1199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:59.589540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:59.589587 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:59.589654 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:59.590600 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.591084 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.592052 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.592262 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.592397 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:59.594506 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:59.594509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:59.594512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:59.594537 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:59.596175 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:59.596665 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:59.596668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:59.596669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:59.596753 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:59.597023 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:59.597025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:59.597027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:59.597212 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:59.597289 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:59.597653 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 14:51:59.597746 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:51:59.598500 24150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:59.587790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:33521 (local address 127.19.34.67:51525)
0619 14:51:59.588209 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:59.588244 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:59.589359 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:59.589812 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:59.589818 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:59.590100 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:59.590478 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:59.590489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.591197 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.591200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:59.591943 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:59.591951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.592424 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.592427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:59.592572 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:59.593035 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:59.593049 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:59.594361 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:59.596277 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:59.596281 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:59.596290 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:59.596560 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:59.596837 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:59.596839 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:59.596840 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:59.596946 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:59.597294 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:59.597298 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:59.597472 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 14:51:59.597599 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 14:51:59.597789 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
I20260619 14:51:59.599694 24148 heartbeater.cc:347] Connected to a master server at 127.19.34.126:33521
I20260619 14:51:59.599900 24148 heartbeater.cc:464] Registering TS with master...
I20260619 14:51:59.600415 24148 heartbeater.cc:511] Master 127.19.34.126:33521 requested a full tablet report, sending...
I20260619 14:51:59.601370 23692 ts_manager.cc:194] Registered new tserver with Master: 659fa6281f474d358859e7cfec6bf2f8 (127.19.34.67:35211)
I20260619 14:51:59.601890 23692 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:51525
I20260619 14:51:59.611784 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:51:59.620146 23731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:59.613538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44054 (local address 127.19.34.126:33521)
0619 14:51:59.613732 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:59.613736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:59.613934 (+ 198us) server_negotiation.cc:407] Connection header received
0619 14:51:59.614066 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:59.614070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:59.614107 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:59.614194 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:59.614836 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.615334 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.616100 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.616362 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.616468 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:59.617637 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:59.617641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:59.617643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:59.617665 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:59.618728 (+ 1063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:59.619090 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:59.619095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:59.619097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:59.619168 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:59.619412 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:59.619417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:59.619418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:59.619601 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:59.619731 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:59.619920 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 14:51:59.620011 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 14:51:59.622431 23692 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:51:59.625286 23692 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:51:59.637238 24162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:59.632529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:35211 (local address 127.0.0.1:33460)
0619 14:51:59.633075 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:59.633087 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:59.633198 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:59.633735 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:59.633738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:59.633753 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:59.633827 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:59.633831 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.635014 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.635020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:59.635774 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:59.635782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.636760 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.636763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:59.636854 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:59.636886 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 14:51:59.636986 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 14:51:59.637096 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
I20260619 14:51:59.637439 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:59.632073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32794 (local address 127.19.34.66:38767)
0619 14:51:59.632484 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:59.632494 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:59.632515 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:51:59.632664 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:59.632668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:59.632847 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:59.633004 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:59.633427 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.634537 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.636130 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.636900 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.637013 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:59.637127 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:59.637203 (+ 76us) server_negotiation.cc:299] Negotiation successful
0619 14:51:59.637299 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":127,"threads_started":1}
I20260619 14:51:59.637575 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:59.632714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33460 (local address 127.19.34.67:35211)
0619 14:51:59.633006 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:59.633017 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:59.633252 (+ 235us) server_negotiation.cc:407] Connection header received
0619 14:51:59.633315 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:59.633320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:59.633484 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:59.633603 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:59.633934 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.634895 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.635886 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.636648 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.636761 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:59.636875 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:59.637329 (+ 454us) server_negotiation.cc:299] Negotiation successful
0619 14:51:59.637422 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":111,"threads_started":1}
I20260619 14:51:59.637470 24159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:59.631932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:38767 (local address 127.0.0.1:32794)
0619 14:51:59.632355 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:59.632398 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:59.632500 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:59.633172 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:59.633179 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:59.633209 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:59.633316 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:59.633321 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.634681 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.634684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:59.636012 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:59.636021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.637012 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.637015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:59.637102 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:59.637119 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 14:51:59.637164 (+ 45us) client_negotiation.cc:240] Negotiation successful
0619 14:51:59.637212 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":38,"thread_start_us":66,"threads_started":1}
I20260619 14:51:59.637873 24163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:59.632512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56846 (local address 127.19.34.65:32971)
0619 14:51:59.632957 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:59.632968 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:59.633166 (+ 198us) server_negotiation.cc:407] Connection header received
0619 14:51:59.633370 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:59.633376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:59.633552 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:59.633689 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:59.634059 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.634908 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.636254 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.636953 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.637033 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:59.637127 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:59.637653 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 14:51:59.637740 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":131,"threads_started":1}
I20260619 14:51:59.638202 24160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:59.632037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:32971 (local address 127.0.0.1:56846)
0619 14:51:59.633059 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:59.633071 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:59.633160 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:59.633837 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:59.633840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:59.633857 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:59.633952 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:59.633957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.635052 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.635055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:59.636152 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:59.636158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.637187 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.637191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:59.637268 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:59.637306 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 14:51:59.638024 (+ 718us) client_negotiation.cc:240] Negotiation successful
0619 14:51:59.638090 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"mutex_wait_us":13,"thread_start_us":72,"threads_started":1}
I20260619 14:51:59.640173 24082 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9035bc7b584074a66cabe192fd32ae (DEFAULT_TABLE table=test-table [id=bfa6007e6d1648ce965240d177417816]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:59.640360 23816 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9035bc7b584074a66cabe192fd32ae (DEFAULT_TABLE table=test-table [id=bfa6007e6d1648ce965240d177417816]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:59.640268 23949 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9035bc7b584074a66cabe192fd32ae (DEFAULT_TABLE table=test-table [id=bfa6007e6d1648ce965240d177417816]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:51:59.641073 24082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9035bc7b584074a66cabe192fd32ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:59.641153 23949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9035bc7b584074a66cabe192fd32ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:59.641155 23816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9035bc7b584074a66cabe192fd32ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:59.646853 24165 tablet_bootstrap.cc:492] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8: Bootstrap starting.
I20260619 14:51:59.648073 24167 tablet_bootstrap.cc:492] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757: Bootstrap starting.
I20260619 14:51:59.648109 24166 tablet_bootstrap.cc:492] T 8b9035bc7b584074a66cabe192fd32ae P 4d7708b9b74846d194d528edf9873229: Bootstrap starting.
I20260619 14:51:59.649397 24165 tablet_bootstrap.cc:654] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:59.650153 24167 tablet_bootstrap.cc:654] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:59.650153 24166 tablet_bootstrap.cc:654] T 8b9035bc7b584074a66cabe192fd32ae P 4d7708b9b74846d194d528edf9873229: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:59.650408 24165 log.cc:824] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:59.651358 24166 log.cc:824] T 8b9035bc7b584074a66cabe192fd32ae P 4d7708b9b74846d194d528edf9873229: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:59.651358 24167 log.cc:824] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:59.652252 24165 tablet_bootstrap.cc:492] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8: No bootstrap required, opened a new log
I20260619 14:51:59.652444 24165 ts_tablet_manager.cc:1397] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:51:59.653038 24167 tablet_bootstrap.cc:492] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757: No bootstrap required, opened a new log
I20260619 14:51:59.653203 24167 ts_tablet_manager.cc:1397] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:51:59.653837 24166 tablet_bootstrap.cc:492] T 8b9035bc7b584074a66cabe192fd32ae P 4d7708b9b74846d194d528edf9873229: No bootstrap required, opened a new log
I20260619 14:51:59.654026 24166 ts_tablet_manager.cc:1397] T 8b9035bc7b584074a66cabe192fd32ae P 4d7708b9b74846d194d528edf9873229: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:51:59.655885 24167 raft_consensus.cc:348] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94a91900e644c249e285df11925d757" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 38767 } } peers { permanent_uuid: "4d7708b9b74846d194d528edf9873229" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 32971 } } peers { permanent_uuid: "659fa6281f474d358859e7cfec6bf2f8" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35211 } }
I20260619 14:51:59.656119 24167 raft_consensus.cc:374] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:59.656167 24167 raft_consensus.cc:729] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94a91900e644c249e285df11925d757, State: Initialized, Role: FOLLOWER
I20260619 14:51:59.656054 24165 raft_consensus.cc:348] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94a91900e644c249e285df11925d757" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 38767 } } peers { permanent_uuid: "4d7708b9b74846d194d528edf9873229" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 32971 } } peers { permanent_uuid: "659fa6281f474d358859e7cfec6bf2f8" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35211 } }
I20260619 14:51:59.656272 24165 raft_consensus.cc:374] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:59.656344 24165 raft_consensus.cc:729] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 659fa6281f474d358859e7cfec6bf2f8, State: Initialized, Role: FOLLOWER
I20260619 14:51:59.656627 24167 consensus_queue.cc:260] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94a91900e644c249e285df11925d757" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 38767 } } peers { permanent_uuid: "4d7708b9b74846d194d528edf9873229" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 32971 } } peers { permanent_uuid: "659fa6281f474d358859e7cfec6bf2f8" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35211 } }
I20260619 14:51:59.656730 24165 consensus_queue.cc:260] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94a91900e644c249e285df11925d757" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 38767 } } peers { permanent_uuid: "4d7708b9b74846d194d528edf9873229" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 32971 } } peers { permanent_uuid: "659fa6281f474d358859e7cfec6bf2f8" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35211 } }
I20260619 14:51:59.657223 24015 heartbeater.cc:503] Master 127.19.34.126:33521 was elected leader, sending a full tablet report...
I20260619 14:51:59.657475 24167 ts_tablet_manager.cc:1428] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 14:51:59.657545 24166 raft_consensus.cc:348] T 8b9035bc7b584074a66cabe192fd32ae P 4d7708b9b74846d194d528edf9873229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94a91900e644c249e285df11925d757" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 38767 } } peers { permanent_uuid: "4d7708b9b74846d194d528edf9873229" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 32971 } } peers { permanent_uuid: "659fa6281f474d358859e7cfec6bf2f8" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35211 } }
I20260619 14:51:59.657764 24166 raft_consensus.cc:374] T 8b9035bc7b584074a66cabe192fd32ae P 4d7708b9b74846d194d528edf9873229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:59.657811 24166 raft_consensus.cc:729] T 8b9035bc7b584074a66cabe192fd32ae P 4d7708b9b74846d194d528edf9873229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d7708b9b74846d194d528edf9873229, State: Initialized, Role: FOLLOWER
I20260619 14:51:59.658159 24166 consensus_queue.cc:260] T 8b9035bc7b584074a66cabe192fd32ae P 4d7708b9b74846d194d528edf9873229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94a91900e644c249e285df11925d757" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 38767 } } peers { permanent_uuid: "4d7708b9b74846d194d528edf9873229" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 32971 } } peers { permanent_uuid: "659fa6281f474d358859e7cfec6bf2f8" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35211 } }
I20260619 14:51:59.658524 24165 ts_tablet_manager.cc:1428] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:51:59.658720 23882 heartbeater.cc:503] Master 127.19.34.126:33521 was elected leader, sending a full tablet report...
I20260619 14:51:59.658785 24148 heartbeater.cc:503] Master 127.19.34.126:33521 was elected leader, sending a full tablet report...
I20260619 14:51:59.658874 24166 ts_tablet_manager.cc:1428] T 8b9035bc7b584074a66cabe192fd32ae P 4d7708b9b74846d194d528edf9873229: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260619 14:51:59.676560 23883 tablet.cc:2369] T 8b9035bc7b584074a66cabe192fd32ae P 4d7708b9b74846d194d528edf9873229: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:51:59.836833 24149 tablet.cc:2369] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:51:59.884137 24016 tablet.cc:2369] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:51:59.907552 24172 raft_consensus.cc:482] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:51:59.907727 24172 raft_consensus.cc:504] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94a91900e644c249e285df11925d757" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 38767 } } peers { permanent_uuid: "4d7708b9b74846d194d528edf9873229" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 32971 } } peers { permanent_uuid: "659fa6281f474d358859e7cfec6bf2f8" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35211 } }
I20260619 14:51:59.908917 24172 leader_election.cc:288] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b94a91900e644c249e285df11925d757 (127.19.34.66:38767), 4d7708b9b74846d194d528edf9873229 (127.19.34.65:32971)
I20260619 14:51:59.913448 24163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:59.909284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:58477 (local address 127.19.34.65:32971)
0619 14:51:59.909417 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:59.909421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:59.909438 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:51:59.909526 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:59.909528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:59.909566 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:59.909624 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:59.909936 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.910679 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.912189 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.912758 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.912890 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:59.912951 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:59.913238 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 14:51:59.913299 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:51:59.913448 24161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:59.909236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:43043 (local address 127.19.34.66:38767)
0619 14:51:59.909352 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:59.909356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:59.909720 (+ 364us) server_negotiation.cc:407] Connection header received
0619 14:51:59.909788 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:59.909790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:59.909823 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:59.909872 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:51:59.910089 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.910753 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.912183 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.912772 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.912890 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:59.912951 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:59.913238 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 14:51:59.913299 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:51:59.913502 24150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:59.909160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:32971 (local address 127.19.34.67:58477)
0619 14:51:59.909295 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:59.909309 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:59.909413 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:59.909660 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:59.909665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:59.909686 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:59.909819 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:59.909829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.910777 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.910779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:59.912052 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:59.912062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.912937 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.912940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:59.913105 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:59.913128 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:51:59.913209 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:51:59.913308 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":35,"spinlock_wait_cycles":1792}
I20260619 14:51:59.913533 24174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:59.909160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:38767 (local address 127.19.34.67:43043)
0619 14:51:59.909598 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:59.909634 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:59.909723 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:59.909907 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:59.909909 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:59.909927 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:51:59.910005 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:59.910009 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.910841 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.910843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:59.912052 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:59.912062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.912937 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.912940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:59.913074 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:59.913128 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 14:51:59.913209 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:51:59.913308 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":190,"thread_start_us":118,"threads_started":1}
I20260619 14:51:59.914186 23836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9035bc7b584074a66cabe192fd32ae" candidate_uuid: "659fa6281f474d358859e7cfec6bf2f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d7708b9b74846d194d528edf9873229" is_pre_election: true
I20260619 14:51:59.914181 23969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9035bc7b584074a66cabe192fd32ae" candidate_uuid: "659fa6281f474d358859e7cfec6bf2f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94a91900e644c249e285df11925d757" is_pre_election: true
I20260619 14:51:59.914503 23836 raft_consensus.cc:2454] T 8b9035bc7b584074a66cabe192fd32ae P 4d7708b9b74846d194d528edf9873229 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 659fa6281f474d358859e7cfec6bf2f8 in term 0.
I20260619 14:51:59.914536 23969 raft_consensus.cc:2454] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 659fa6281f474d358859e7cfec6bf2f8 in term 0.
I20260619 14:51:59.915009 24037 leader_election.cc:302] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4d7708b9b74846d194d528edf9873229, 659fa6281f474d358859e7cfec6bf2f8, b94a91900e644c249e285df11925d757; no voters:
I20260619 14:51:59.915283 24172 raft_consensus.cc:2781] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:51:59.915367 24172 raft_consensus.cc:482] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:51:59.915398 24172 raft_consensus.cc:3037] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:59.916555 24172 raft_consensus.cc:504] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94a91900e644c249e285df11925d757" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 38767 } } peers { permanent_uuid: "4d7708b9b74846d194d528edf9873229" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 32971 } } peers { permanent_uuid: "659fa6281f474d358859e7cfec6bf2f8" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35211 } }
I20260619 14:51:59.916940 24172 leader_election.cc:288] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 [CANDIDATE]: Term 1 election: Requested vote from peers b94a91900e644c249e285df11925d757 (127.19.34.66:38767), 4d7708b9b74846d194d528edf9873229 (127.19.34.65:32971)
I20260619 14:51:59.917277 23969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9035bc7b584074a66cabe192fd32ae" candidate_uuid: "659fa6281f474d358859e7cfec6bf2f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94a91900e644c249e285df11925d757"
I20260619 14:51:59.917315 23836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9035bc7b584074a66cabe192fd32ae" candidate_uuid: "659fa6281f474d358859e7cfec6bf2f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d7708b9b74846d194d528edf9873229"
I20260619 14:51:59.917433 23836 raft_consensus.cc:3037] T 8b9035bc7b584074a66cabe192fd32ae P 4d7708b9b74846d194d528edf9873229 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:59.917426 23969 raft_consensus.cc:3037] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:59.918365 23836 raft_consensus.cc:2454] T 8b9035bc7b584074a66cabe192fd32ae P 4d7708b9b74846d194d528edf9873229 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 659fa6281f474d358859e7cfec6bf2f8 in term 1.
I20260619 14:51:59.918423 23969 raft_consensus.cc:2454] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 659fa6281f474d358859e7cfec6bf2f8 in term 1.
I20260619 14:51:59.918671 24037 leader_election.cc:302] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 659fa6281f474d358859e7cfec6bf2f8, b94a91900e644c249e285df11925d757; no voters:
I20260619 14:51:59.918848 24172 raft_consensus.cc:2781] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:59.919059 24172 raft_consensus.cc:686] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 [term 1 LEADER]: Becoming Leader. State: Replica: 659fa6281f474d358859e7cfec6bf2f8, State: Running, Role: LEADER
I20260619 14:51:59.919370 24172 consensus_queue.cc:237] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94a91900e644c249e285df11925d757" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 38767 } } peers { permanent_uuid: "4d7708b9b74846d194d528edf9873229" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 32971 } } peers { permanent_uuid: "659fa6281f474d358859e7cfec6bf2f8" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35211 } }
I20260619 14:51:59.922235 23691 catalog_manager.cc:5495] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 659fa6281f474d358859e7cfec6bf2f8 (127.19.34.67). New cstate: current_term: 1 leader_uuid: "659fa6281f474d358859e7cfec6bf2f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b94a91900e644c249e285df11925d757" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 38767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d7708b9b74846d194d528edf9873229" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 32971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "659fa6281f474d358859e7cfec6bf2f8" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35211 } health_report { overall_health: HEALTHY } } }
I20260619 14:51:59.996112 24164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:59.992647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33476 (local address 127.19.34.67:35211)
0619 14:51:59.992779 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:59.992783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:59.992799 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:51:59.992881 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:59.992883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:59.992930 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:59.993049 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:59.993286 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.993811 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.994674 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:59.994897 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:59.995007 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:59.995313 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:59.995751 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:59.995871 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:59.995923 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 14:51:59.995973 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:52:00.001811 23836 raft_consensus.cc:1261] T 8b9035bc7b584074a66cabe192fd32ae P 4d7708b9b74846d194d528edf9873229 [term 1 FOLLOWER]: Refusing update from remote peer 659fa6281f474d358859e7cfec6bf2f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:00.001822 23969 raft_consensus.cc:1261] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757 [term 1 FOLLOWER]: Refusing update from remote peer 659fa6281f474d358859e7cfec6bf2f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:00.002485 24172 consensus_queue.cc:1035] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b94a91900e644c249e285df11925d757" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 38767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:00.002692 24175 consensus_queue.cc:1035] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d7708b9b74846d194d528edf9873229" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 32971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:00.011058 24180 mvcc.cc:205] Tried to move back new op lower bound from 7298583429120090112 to 7298583428792180736. Current Snapshot: MvccSnapshot[applied={T|T < 7298583429120090112 or (T in {7298583429120090112})}]
I20260619 14:52:00.014189 24178 mvcc.cc:205] Tried to move back new op lower bound from 7298583429120090112 to 7298583428792180736. Current Snapshot: MvccSnapshot[applied={T|T < 7298583429120090112 or (T in {7298583429120090112})}]
I20260619 14:52:00.017910 23691 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:00.018041 23691 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:00.020110 23691 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d923bb0b54d249f488bf5ce4b381e300: Sending DeleteTablet for 3 replicas of tablet 8b9035bc7b584074a66cabe192fd32ae
I20260619 14:52:00.020849 23816 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b9035bc7b584074a66cabe192fd32ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:00 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:56846
I20260619 14:52:00.020936 23949 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b9035bc7b584074a66cabe192fd32ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:00 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:32794
I20260619 14:52:00.021025 24082 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b9035bc7b584074a66cabe192fd32ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:00 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:33460
I20260619 14:52:00.021476 24189 tablet_replica.cc:330] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757: stopping tablet replica
I20260619 14:52:00.021673 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 23754
I20260619 14:52:00.021823 24189 raft_consensus.cc:2229] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:00.022434 24189 raft_consensus.cc:2258] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:00.023471 24189 ts_tablet_manager.cc:1905] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:00.026151 24189 ts_tablet_manager.cc:1918] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:00.026265 24189 log.cc:1196] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-1/wal/wals/8b9035bc7b584074a66cabe192fd32ae
I20260619 14:52:00.026625 24189 ts_tablet_manager.cc:1939] T 8b9035bc7b584074a66cabe192fd32ae P b94a91900e644c249e285df11925d757: Deleting consensus metadata
I20260619 14:52:00.027741 23678 catalog_manager.cc:4841] TS b94a91900e644c249e285df11925d757 (127.19.34.66:38767): tablet 8b9035bc7b584074a66cabe192fd32ae (table test-table [id=bfa6007e6d1648ce965240d177417816]) successfully deleted
I20260619 14:52:00.028457 24190 tablet_replica.cc:330] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8: stopping tablet replica
I20260619 14:52:00.028767 24190 raft_consensus.cc:2229] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:52:00.029084 24190 raft_consensus.cc:2258] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:00.030419 24190 ts_tablet_manager.cc:1905] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 14:52:00.030500 23677 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.34.65:32971 (error 108)
I20260619 14:52:00.030990 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 23887
I20260619 14:52:00.032631 24190 ts_tablet_manager.cc:1918] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:00.032732 24190 log.cc:1196] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/8b9035bc7b584074a66cabe192fd32ae
I20260619 14:52:00.033004 24190 ts_tablet_manager.cc:1939] T 8b9035bc7b584074a66cabe192fd32ae P 659fa6281f474d358859e7cfec6bf2f8: Deleting consensus metadata
I20260619 14:52:00.033926 23678 catalog_manager.cc:4841] TS 659fa6281f474d358859e7cfec6bf2f8 (127.19.34.67:35211): tablet 8b9035bc7b584074a66cabe192fd32ae (table test-table [id=bfa6007e6d1648ce965240d177417816]) successfully deleted
I20260619 14:52:00.036728 24160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:00.036487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:32971 (local address 127.0.0.1:56856)
0619 14:52:00.036598 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:00.036667 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.65:32971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 14:52:00.036911 23677 catalog_manager.cc:4584] TS 4d7708b9b74846d194d528edf9873229 (127.19.34.65:32971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b9035bc7b584074a66cabe192fd32ae: Network error: Client connection negotiation failed: client connection to 127.19.34.65:32971: connect: Connection refused (error 111)
I20260619 14:52:00.038602 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 24020
I20260619 14:52:00.042750 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 23661
2026-06-19T14:52:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2348 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T14:52:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:52:00Z Disabled control of system clock
I20260619 14:52:00.070284 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:43785
--webserver_interface=127.19.34.126
--webserver_port=0
--builtin_ntp_servers=127.19.34.84:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:43785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 14:52:00.167174 24195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:00.167382 24195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:00.167413 24195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:00.171100 24195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:00.171146 24195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:00.171162 24195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:00.171206 24195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:00.171238 24195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:52:00.175410 24195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:43785
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:43785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:00.176183 24195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:00.177268 24195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:00.184288 24201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:00.184294 24203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:00.184417 24195 server_base.cc:1034] running on GCE node
W20260619 14:52:00.184567 24200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:00.185139 24195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:00.186029 24195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:00.187187 24195 hybrid_clock.cc:648] HybridClock initialized: now 1781880720187161 us; error 37 us; skew 500 ppm
I20260619 14:52:00.188925 24195 webserver.cc:460] Webserver started at http://127.19.34.126:35921/ using document root <none> and password file <none>
I20260619 14:52:00.189443 24195 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:00.189505 24195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:00.189865 24195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:00.191524 24195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "9e6f86d9cc2d4b8d88cee1825caed930"
format_stamp: "Formatted at 2026-06-19 14:52:00 on dist-test-slave-3qdh"
server_key: "139d21b9daa21b2ea8e041d7c3a13761"
server_key_iv: "9477310427865a5d172d457e6af072ce"
server_key_version: "encryptionkey@0"
I20260619 14:52:00.192010 24195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "9e6f86d9cc2d4b8d88cee1825caed930"
format_stamp: "Formatted at 2026-06-19 14:52:00 on dist-test-slave-3qdh"
server_key: "139d21b9daa21b2ea8e041d7c3a13761"
server_key_iv: "9477310427865a5d172d457e6af072ce"
server_key_version: "encryptionkey@0"
I20260619 14:52:00.195134 24195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 14:52:00.197089 24209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:00.198096 24195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:00.198230 24195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "9e6f86d9cc2d4b8d88cee1825caed930"
format_stamp: "Formatted at 2026-06-19 14:52:00 on dist-test-slave-3qdh"
server_key: "139d21b9daa21b2ea8e041d7c3a13761"
server_key_iv: "9477310427865a5d172d457e6af072ce"
server_key_version: "encryptionkey@0"
I20260619 14:52:00.198328 24195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:00.202162 24195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:00.202620 24195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:00.202781 24195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:00.208696 24195 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:43785
I20260619 14:52:00.208700 24261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:43785 every 8 connection(s)
I20260619 14:52:00.209415 24195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:52:00.211835 24262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:00.214481 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 24195
I20260619 14:52:00.214614 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:52:00.214852 19593 external_mini_cluster.cc:1428] Setting key 39b70b93f088310482ca6bfde98b1d4b
I20260619 14:52:00.216738 24262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930: Bootstrap starting.
I20260619 14:52:00.218629 24262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:00.219310 24262 log.cc:824] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:00.219547 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:00.216033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56580 (local address 127.19.34.126:43785)
0619 14:52:00.216617 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:00.216630 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:00.216684 (+ 54us) server_negotiation.cc:407] Connection header received
0619 14:52:00.217254 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:00.217269 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:00.217550 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:00.217842 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:00.218523 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:00.218527 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:00.218536 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:00.218546 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:00.218591 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:00.218604 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:00.218704 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:00.218799 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 14:52:00.218995 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":102,"threads_started":1}
I20260619 14:52:00.220996 24262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930: No bootstrap required, opened a new log
I20260619 14:52:00.224033 24262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e6f86d9cc2d4b8d88cee1825caed930" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43785 } }
I20260619 14:52:00.224249 24262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:00.224349 24262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e6f86d9cc2d4b8d88cee1825caed930, State: Initialized, Role: FOLLOWER
I20260619 14:52:00.224798 24262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e6f86d9cc2d4b8d88cee1825caed930" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43785 } }
I20260619 14:52:00.224968 24262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:52:00.225036 24262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:52:00.225126 24262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:00.225999 24262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e6f86d9cc2d4b8d88cee1825caed930" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43785 } }
I20260619 14:52:00.226282 24262 leader_election.cc:302] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e6f86d9cc2d4b8d88cee1825caed930; no voters:
I20260619 14:52:00.226536 24262 leader_election.cc:288] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:52:00.226648 24267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:00.226872 24267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930 [term 1 LEADER]: Becoming Leader. State: Replica: 9e6f86d9cc2d4b8d88cee1825caed930, State: Running, Role: LEADER
I20260619 14:52:00.227121 24267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e6f86d9cc2d4b8d88cee1825caed930" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43785 } }
I20260619 14:52:00.227545 24262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:00.228615 24269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e6f86d9cc2d4b8d88cee1825caed930. Latest consensus state: current_term: 1 leader_uuid: "9e6f86d9cc2d4b8d88cee1825caed930" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e6f86d9cc2d4b8d88cee1825caed930" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43785 } } }
I20260619 14:52:00.228725 24269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:00.228991 24268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e6f86d9cc2d4b8d88cee1825caed930" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e6f86d9cc2d4b8d88cee1825caed930" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43785 } } }
I20260619 14:52:00.229089 24268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:00.229164 24276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:52:00.231220 24276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:52:00.235347 24276 catalog_manager.cc:1347] Generated new cluster ID: 95d9ea9089e444e89a175c7ee2a9e9ec
I20260619 14:52:00.235404 24276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:52:00.247485 24276 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:52:00.247720 24276 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:52:00.252351 24276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930: Generated new TSK 0
I20260619 14:52:00.252923 24276 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:52:00.256950 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:43785
--builtin_ntp_servers=127.19.34.84:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 14:52:00.354673 24286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:00.354888 24286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:00.354926 24286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:00.358671 24286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:00.358718 24286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:00.358810 24286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:52:00.363005 24286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:43785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:00.363867 24286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:00.365116 24286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:00.373497 24292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:00.373571 24286 server_base.cc:1034] running on GCE node
W20260619 14:52:00.373546 24291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:00.373517 24294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:00.374015 24286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:00.374519 24286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:00.375660 24286 hybrid_clock.cc:648] HybridClock initialized: now 1781880720375639 us; error 38 us; skew 500 ppm
I20260619 14:52:00.377194 24286 webserver.cc:460] Webserver started at http://127.19.34.65:40435/ using document root <none> and password file <none>
I20260619 14:52:00.377686 24286 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:00.377739 24286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:00.377921 24286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:00.379635 24286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "0385489d421847c287073da102281bde"
format_stamp: "Formatted at 2026-06-19 14:52:00 on dist-test-slave-3qdh"
server_key: "a8339bf95b78ce6bb90ddfa780da0f3c"
server_key_iv: "7d85807ac0aec4f824cf39dcc509b812"
server_key_version: "encryptionkey@0"
I20260619 14:52:00.380111 24286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "0385489d421847c287073da102281bde"
format_stamp: "Formatted at 2026-06-19 14:52:00 on dist-test-slave-3qdh"
server_key: "a8339bf95b78ce6bb90ddfa780da0f3c"
server_key_iv: "7d85807ac0aec4f824cf39dcc509b812"
server_key_version: "encryptionkey@0"
I20260619 14:52:00.383216 24286 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:52:00.385401 24300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:00.386435 24286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:52:00.386541 24286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "0385489d421847c287073da102281bde"
format_stamp: "Formatted at 2026-06-19 14:52:00 on dist-test-slave-3qdh"
server_key: "a8339bf95b78ce6bb90ddfa780da0f3c"
server_key_iv: "7d85807ac0aec4f824cf39dcc509b812"
server_key_version: "encryptionkey@0"
I20260619 14:52:00.386632 24286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:00.390494 24286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:00.391103 24286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:00.391250 24286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:00.391820 24286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:00.392867 24286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:00.392922 24286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:00.392971 24286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:00.393003 24286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:00.402325 24286 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:43215
I20260619 14:52:00.402347 24413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:43215 every 8 connection(s)
I20260619 14:52:00.403007 24286 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:52:00.407855 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:00.404481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:36447 (local address 127.19.34.126:43785)
0619 14:52:00.404610 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:00.404613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:00.405553 (+ 940us) server_negotiation.cc:407] Connection header received
0619 14:52:00.406619 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:00.406621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:00.406653 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:00.406719 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:00.407519 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:00.407521 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:00.407523 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:00.407533 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:00.407550 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:00.407557 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:00.407620 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:00.407723 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 14:52:00.407754 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:52:00.408595 24416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:00.404829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43785 (local address 127.19.34.65:36447)
0619 14:52:00.405406 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:00.405453 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:00.406458 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:00.406843 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:00.406849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:00.406916 (+ 67us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:00.407323 (+ 407us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:52:00.407324 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:00.407354 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:52:00.407404 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:00.407621 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:00.407626 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:00.407629 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:52:00.407736 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 14:52:00.407906 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":110,"threads_started":1}
I20260619 14:52:00.409614 24414 heartbeater.cc:347] Connected to a master server at 127.19.34.126:43785
I20260619 14:52:00.409792 24414 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:00.410131 24414 heartbeater.cc:511] Master 127.19.34.126:43785 requested a full tablet report, sending...
I20260619 14:52:00.411314 24226 ts_manager.cc:194] Registered new tserver with Master: 0385489d421847c287073da102281bde (127.19.34.65:43215)
I20260619 14:52:00.411969 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 24286
I20260619 14:52:00.412066 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:52:00.412356 19593 external_mini_cluster.cc:1428] Setting key 8219b1d37152e4419327f58daaf02516
I20260619 14:52:00.413834 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:43785
--builtin_ntp_servers=127.19.34.84:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 14:52:00.507988 24417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:00.508193 24417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:00.508226 24417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:00.511550 24417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:00.511588 24417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:00.511632 24417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:52:00.515766 24417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:43785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:00.516535 24417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:00.517573 24417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:52:00.525358 24417 server_base.cc:1034] running on GCE node
W20260619 14:52:00.525312 24422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:00.525305 24423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:00.525312 24425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:00.525885 24417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:00.526436 24417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:00.527602 24417 hybrid_clock.cc:648] HybridClock initialized: now 1781880720527586 us; error 30 us; skew 500 ppm
I20260619 14:52:00.529232 24417 webserver.cc:460] Webserver started at http://127.19.34.66:44145/ using document root <none> and password file <none>
I20260619 14:52:00.529677 24417 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:00.529753 24417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:00.529922 24417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:00.531559 24417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "eb61a0a2359447b1a917c79091fa4232"
format_stamp: "Formatted at 2026-06-19 14:52:00 on dist-test-slave-3qdh"
server_key: "c9656a92786551b292f1a182d34a9352"
server_key_iv: "05b6f8d0a46497c530c7ab676742f25d"
server_key_version: "encryptionkey@0"
I20260619 14:52:00.532009 24417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "eb61a0a2359447b1a917c79091fa4232"
format_stamp: "Formatted at 2026-06-19 14:52:00 on dist-test-slave-3qdh"
server_key: "c9656a92786551b292f1a182d34a9352"
server_key_iv: "05b6f8d0a46497c530c7ab676742f25d"
server_key_version: "encryptionkey@0"
I20260619 14:52:00.535035 24417 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:52:00.537139 24431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:00.538136 24417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:00.538295 24417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "eb61a0a2359447b1a917c79091fa4232"
format_stamp: "Formatted at 2026-06-19 14:52:00 on dist-test-slave-3qdh"
server_key: "c9656a92786551b292f1a182d34a9352"
server_key_iv: "05b6f8d0a46497c530c7ab676742f25d"
server_key_version: "encryptionkey@0"
I20260619 14:52:00.538401 24417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:00.542352 24417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:00.542834 24417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:00.542992 24417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:00.543495 24417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:00.544260 24417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:00.544361 24417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:00.544425 24417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:00.544476 24417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:00.551946 24417 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:34735
I20260619 14:52:00.551970 24544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:34735 every 8 connection(s)
I20260619 14:52:00.552735 24417 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:52:00.557886 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:00.554218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:53753 (local address 127.19.34.126:43785)
0619 14:52:00.554365 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:00.554368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:00.555225 (+ 857us) server_negotiation.cc:407] Connection header received
0619 14:52:00.556392 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:00.556396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:00.556438 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:00.556508 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:00.557405 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:00.557408 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:00.557410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:00.557421 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:00.557457 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:00.557471 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:00.557561 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:00.557715 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 14:52:00.557758 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:52:00.558550 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 24417
I20260619 14:52:00.558669 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:52:00.558717 24547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:00.554514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43785 (local address 127.19.34.66:53753)
0619 14:52:00.555082 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:00.555115 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:00.556163 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:00.556633 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:00.556639 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:00.556710 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:00.557177 (+ 467us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:52:00.557184 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:00.557214 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:52:00.557266 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:00.557557 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:00.557565 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:00.557573 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:52:00.557738 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 14:52:00.557941 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":115,"threads_started":1}
I20260619 14:52:00.558935 19593 external_mini_cluster.cc:1428] Setting key e34f40b8524f7b98b8db8ba8f960b978
I20260619 14:52:00.559670 24545 heartbeater.cc:347] Connected to a master server at 127.19.34.126:43785
I20260619 14:52:00.559846 24545 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:00.560148 24545 heartbeater.cc:511] Master 127.19.34.126:43785 requested a full tablet report, sending...
I20260619 14:52:00.560586 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:43785
--builtin_ntp_servers=127.19.34.84:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 14:52:00.561059 24226 ts_manager.cc:194] Registered new tserver with Master: eb61a0a2359447b1a917c79091fa4232 (127.19.34.66:34735)
W20260619 14:52:00.656452 24548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:00.656634 24548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:00.656673 24548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:00.659972 24548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:00.660009 24548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:00.660065 24548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:52:00.664103 24548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:43785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:00.664948 24548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:00.666024 24548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:00.673760 24553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:00.673766 24554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:00.673919 24548 server_base.cc:1034] running on GCE node
W20260619 14:52:00.673874 24556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:00.674492 24548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:00.675053 24548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:00.676373 24548 hybrid_clock.cc:648] HybridClock initialized: now 1781880720676360 us; error 36 us; skew 500 ppm
I20260619 14:52:00.677783 24548 webserver.cc:460] Webserver started at http://127.19.34.67:36931/ using document root <none> and password file <none>
I20260619 14:52:00.678228 24548 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:00.678305 24548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:00.678489 24548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:00.680043 24548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "ad8d4b19165f44ea99a8039371c6228f"
format_stamp: "Formatted at 2026-06-19 14:52:00 on dist-test-slave-3qdh"
server_key: "04fe7d60d20df1e16bba41a1dbc866b1"
server_key_iv: "137511990959712d3e6ed612cc0aa887"
server_key_version: "encryptionkey@0"
I20260619 14:52:00.680505 24548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "ad8d4b19165f44ea99a8039371c6228f"
format_stamp: "Formatted at 2026-06-19 14:52:00 on dist-test-slave-3qdh"
server_key: "04fe7d60d20df1e16bba41a1dbc866b1"
server_key_iv: "137511990959712d3e6ed612cc0aa887"
server_key_version: "encryptionkey@0"
I20260619 14:52:00.683350 24548 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260619 14:52:00.685417 24562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:00.686446 24548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:52:00.686565 24548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "ad8d4b19165f44ea99a8039371c6228f"
format_stamp: "Formatted at 2026-06-19 14:52:00 on dist-test-slave-3qdh"
server_key: "04fe7d60d20df1e16bba41a1dbc866b1"
server_key_iv: "137511990959712d3e6ed612cc0aa887"
server_key_version: "encryptionkey@0"
I20260619 14:52:00.686657 24548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:00.690366 24548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:00.690835 24548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:00.690985 24548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:00.691398 24548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:00.692160 24548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:00.692220 24548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:00.692276 24548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:00.692324 24548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:00.699657 24548 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:34957
I20260619 14:52:00.699689 24675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:34957 every 8 connection(s)
I20260619 14:52:00.700408 24548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:52:00.705698 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:00.701907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:40877 (local address 127.19.34.126:43785)
0619 14:52:00.702054 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:00.702059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:00.703043 (+ 984us) server_negotiation.cc:407] Connection header received
0619 14:52:00.704099 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:00.704101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:00.704133 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:00.704199 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:00.705305 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:00.705308 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:00.705310 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:00.705319 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:00.705336 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:00.705344 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:00.705415 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:00.705537 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 14:52:00.705578 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:52:00.705868 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 24548
I20260619 14:52:00.705982 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:52:00.706241 19593 external_mini_cluster.cc:1428] Setting key 2ed4574af827dbcb41906b8bf1e24c9b
I20260619 14:52:00.706524 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:00.702321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43785 (local address 127.19.34.67:40877)
0619 14:52:00.702921 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:00.702952 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:00.703945 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:00.704425 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:00.704431 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:00.704509 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:00.705071 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:52:00.705072 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:00.705106 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:52:00.705181 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:00.705420 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:00.705425 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:00.705429 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:52:00.705566 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 14:52:00.705712 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":113,"threads_started":1}
I20260619 14:52:00.707496 24676 heartbeater.cc:347] Connected to a master server at 127.19.34.126:43785
I20260619 14:52:00.707708 24676 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:00.708117 24676 heartbeater.cc:511] Master 127.19.34.126:43785 requested a full tablet report, sending...
I20260619 14:52:00.709033 24226 ts_manager.cc:194] Registered new tserver with Master: ad8d4b19165f44ea99a8039371c6228f (127.19.34.67:34957)
I20260619 14:52:00.719704 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:52:00.723088 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:00.721411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.19.34.126:43785)
0619 14:52:00.721591 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:00.721595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:00.721781 (+ 186us) server_negotiation.cc:407] Connection header received
0619 14:52:00.721898 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:00.721903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:00.721942 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:00.722029 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:00.722730 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:00.722733 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:00.722734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:00.722760 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:00.722776 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:00.722782 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:00.722857 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:00.722929 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 14:52:00.722955 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 14:52:00.725401 24226 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:52:00.727953 24226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:52:00.737905 24684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:00.734807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:34735 (local address 127.0.0.1:50680)
0619 14:52:00.735091 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:00.735133 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:00.735225 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:00.736517 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:00.736522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:00.736588 (+ 66us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:00.737117 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:52:00.737121 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:00.737144 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:52:00.737220 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:00.737600 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:00.737610 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:00.737617 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:52:00.737722 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 14:52:00.737789 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260619 14:52:00.737948 24686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:00.735203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:34957 (local address 127.0.0.1:49280)
0619 14:52:00.736058 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:00.736071 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:00.736158 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:00.736597 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:00.736600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:00.736615 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:00.737310 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:52:00.737312 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:00.737322 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:52:00.737343 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:00.737738 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:00.737741 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:00.737743 (+ 2us) client_negotiation.cc:769] Sending connection context
0619 14:52:00.737816 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 14:52:00.737847 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":58,"threads_started":1}
I20260619 14:52:00.738319 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:00.734918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50680 (local address 127.19.34.66:34735)
0619 14:52:00.735354 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:00.735383 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:00.735410 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:52:00.735472 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:00.735476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:00.735667 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:00.735802 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:00.737357 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:00.737363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:00.737374 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:00.737385 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:00.737424 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:00.737452 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:00.737557 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:00.738069 (+ 512us) server_negotiation.cc:299] Negotiation successful
0619 14:52:00.738142 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":124,"threads_started":1}
I20260619 14:52:00.738451 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:00.735421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:43215 (local address 127.0.0.1:50120)
0619 14:52:00.736340 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:00.736352 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:00.736414 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:00.737333 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:00.737336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:00.737349 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:00.737714 (+ 365us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:52:00.737717 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:00.737727 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:52:00.737748 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:00.738234 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:00.738237 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:00.738240 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:52:00.738306 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 14:52:00.738340 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":87,"threads_started":1}
I20260619 14:52:00.738498 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:00.735574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.19.34.67:34957)
0619 14:52:00.735942 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:00.735953 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:00.736138 (+ 185us) server_negotiation.cc:407] Connection header received
0619 14:52:00.736240 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:00.736246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:00.736441 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:00.736561 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:00.737489 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:00.737496 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:00.737507 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:00.737517 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:00.737560 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:00.737578 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:00.737698 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:00.738282 (+ 584us) server_negotiation.cc:299] Negotiation successful
0619 14:52:00.738347 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":78,"threads_started":1}
I20260619 14:52:00.738804 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:00.735608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50120 (local address 127.19.34.65:43215)
0619 14:52:00.735989 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:00.736000 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:00.736442 (+ 442us) server_negotiation.cc:407] Connection header received
0619 14:52:00.736502 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:00.736507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:00.736670 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:00.737241 (+ 571us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:00.738017 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:00.738024 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:00.738034 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:00.738045 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:00.738079 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:00.738101 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:00.738195 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:00.738595 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 14:52:00.738671 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":104,"threads_started":1}
I20260619 14:52:00.740986 24610 tablet_service.cc:1467] Processing CreateTablet for tablet c9e4e8a4ed5842d39cea60fc23c271ad (DEFAULT_TABLE table=test-table [id=d9cf62d9b3364addba4e582f404e49d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:00.740984 24348 tablet_service.cc:1467] Processing CreateTablet for tablet c9e4e8a4ed5842d39cea60fc23c271ad (DEFAULT_TABLE table=test-table [id=d9cf62d9b3364addba4e582f404e49d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:00.740985 24479 tablet_service.cc:1467] Processing CreateTablet for tablet c9e4e8a4ed5842d39cea60fc23c271ad (DEFAULT_TABLE table=test-table [id=d9cf62d9b3364addba4e582f404e49d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:00.741925 24610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e4e8a4ed5842d39cea60fc23c271ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:00.741930 24479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e4e8a4ed5842d39cea60fc23c271ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:00.741930 24348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e4e8a4ed5842d39cea60fc23c271ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:00.748065 24690 tablet_bootstrap.cc:492] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f: Bootstrap starting.
I20260619 14:52:00.748955 24691 tablet_bootstrap.cc:492] T c9e4e8a4ed5842d39cea60fc23c271ad P eb61a0a2359447b1a917c79091fa4232: Bootstrap starting.
I20260619 14:52:00.750011 24690 tablet_bootstrap.cc:654] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:00.750546 24691 tablet_bootstrap.cc:654] T c9e4e8a4ed5842d39cea60fc23c271ad P eb61a0a2359447b1a917c79091fa4232: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:00.750715 24690 log.cc:824] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:00.751219 24692 tablet_bootstrap.cc:492] T c9e4e8a4ed5842d39cea60fc23c271ad P 0385489d421847c287073da102281bde: Bootstrap starting.
I20260619 14:52:00.751293 24691 log.cc:824] T c9e4e8a4ed5842d39cea60fc23c271ad P eb61a0a2359447b1a917c79091fa4232: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:00.752506 24690 tablet_bootstrap.cc:492] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f: No bootstrap required, opened a new log
I20260619 14:52:00.752692 24690 ts_tablet_manager.cc:1397] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:52:00.753157 24692 tablet_bootstrap.cc:654] T c9e4e8a4ed5842d39cea60fc23c271ad P 0385489d421847c287073da102281bde: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:00.753190 24691 tablet_bootstrap.cc:492] T c9e4e8a4ed5842d39cea60fc23c271ad P eb61a0a2359447b1a917c79091fa4232: No bootstrap required, opened a new log
I20260619 14:52:00.753333 24691 ts_tablet_manager.cc:1397] T c9e4e8a4ed5842d39cea60fc23c271ad P eb61a0a2359447b1a917c79091fa4232: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:52:00.753872 24692 log.cc:824] T c9e4e8a4ed5842d39cea60fc23c271ad P 0385489d421847c287073da102281bde: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:00.755350 24692 tablet_bootstrap.cc:492] T c9e4e8a4ed5842d39cea60fc23c271ad P 0385489d421847c287073da102281bde: No bootstrap required, opened a new log
I20260619 14:52:00.755556 24692 ts_tablet_manager.cc:1397] T c9e4e8a4ed5842d39cea60fc23c271ad P 0385489d421847c287073da102281bde: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 14:52:00.757016 24690 raft_consensus.cc:348] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb61a0a2359447b1a917c79091fa4232" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34735 } } peers { permanent_uuid: "0385489d421847c287073da102281bde" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 43215 } } peers { permanent_uuid: "ad8d4b19165f44ea99a8039371c6228f" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 34957 } }
I20260619 14:52:00.757344 24690 raft_consensus.cc:374] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:00.757443 24690 raft_consensus.cc:729] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad8d4b19165f44ea99a8039371c6228f, State: Initialized, Role: FOLLOWER
I20260619 14:52:00.757536 24691 raft_consensus.cc:348] T c9e4e8a4ed5842d39cea60fc23c271ad P eb61a0a2359447b1a917c79091fa4232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb61a0a2359447b1a917c79091fa4232" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34735 } } peers { permanent_uuid: "0385489d421847c287073da102281bde" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 43215 } } peers { permanent_uuid: "ad8d4b19165f44ea99a8039371c6228f" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 34957 } }
I20260619 14:52:00.757799 24691 raft_consensus.cc:374] T c9e4e8a4ed5842d39cea60fc23c271ad P eb61a0a2359447b1a917c79091fa4232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:00.757879 24691 raft_consensus.cc:729] T c9e4e8a4ed5842d39cea60fc23c271ad P eb61a0a2359447b1a917c79091fa4232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb61a0a2359447b1a917c79091fa4232, State: Initialized, Role: FOLLOWER
I20260619 14:52:00.757927 24690 consensus_queue.cc:260] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb61a0a2359447b1a917c79091fa4232" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34735 } } peers { permanent_uuid: "0385489d421847c287073da102281bde" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 43215 } } peers { permanent_uuid: "ad8d4b19165f44ea99a8039371c6228f" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 34957 } }
I20260619 14:52:00.758287 24691 consensus_queue.cc:260] T c9e4e8a4ed5842d39cea60fc23c271ad P eb61a0a2359447b1a917c79091fa4232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb61a0a2359447b1a917c79091fa4232" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34735 } } peers { permanent_uuid: "0385489d421847c287073da102281bde" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 43215 } } peers { permanent_uuid: "ad8d4b19165f44ea99a8039371c6228f" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 34957 } }
I20260619 14:52:00.758523 24676 heartbeater.cc:503] Master 127.19.34.126:43785 was elected leader, sending a full tablet report...
I20260619 14:52:00.759032 24692 raft_consensus.cc:348] T c9e4e8a4ed5842d39cea60fc23c271ad P 0385489d421847c287073da102281bde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb61a0a2359447b1a917c79091fa4232" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34735 } } peers { permanent_uuid: "0385489d421847c287073da102281bde" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 43215 } } peers { permanent_uuid: "ad8d4b19165f44ea99a8039371c6228f" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 34957 } }
I20260619 14:52:00.759208 24691 ts_tablet_manager.cc:1428] T c9e4e8a4ed5842d39cea60fc23c271ad P eb61a0a2359447b1a917c79091fa4232: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260619 14:52:00.759222 24692 raft_consensus.cc:374] T c9e4e8a4ed5842d39cea60fc23c271ad P 0385489d421847c287073da102281bde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:00.759267 24692 raft_consensus.cc:729] T c9e4e8a4ed5842d39cea60fc23c271ad P 0385489d421847c287073da102281bde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0385489d421847c287073da102281bde, State: Initialized, Role: FOLLOWER
I20260619 14:52:00.759519 24545 heartbeater.cc:503] Master 127.19.34.126:43785 was elected leader, sending a full tablet report...
I20260619 14:52:00.759668 24692 consensus_queue.cc:260] T c9e4e8a4ed5842d39cea60fc23c271ad P 0385489d421847c287073da102281bde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb61a0a2359447b1a917c79091fa4232" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34735 } } peers { permanent_uuid: "0385489d421847c287073da102281bde" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 43215 } } peers { permanent_uuid: "ad8d4b19165f44ea99a8039371c6228f" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 34957 } }
I20260619 14:52:00.759842 24690 ts_tablet_manager.cc:1428] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:52:00.760496 24692 ts_tablet_manager.cc:1428] T c9e4e8a4ed5842d39cea60fc23c271ad P 0385489d421847c287073da102281bde: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:52:00.760768 24414 heartbeater.cc:503] Master 127.19.34.126:43785 was elected leader, sending a full tablet report...
W20260619 14:52:00.803530 24546 tablet.cc:2369] T c9e4e8a4ed5842d39cea60fc23c271ad P eb61a0a2359447b1a917c79091fa4232: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:52:00.903961 24415 tablet.cc:2369] T c9e4e8a4ed5842d39cea60fc23c271ad P 0385489d421847c287073da102281bde: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:52:00.951365 24677 tablet.cc:2369] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:52:00.957660 24696 raft_consensus.cc:482] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:52:00.957912 24696 raft_consensus.cc:504] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb61a0a2359447b1a917c79091fa4232" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34735 } } peers { permanent_uuid: "0385489d421847c287073da102281bde" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 43215 } } peers { permanent_uuid: "ad8d4b19165f44ea99a8039371c6228f" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 34957 } }
I20260619 14:52:00.958968 24696 leader_election.cc:288] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb61a0a2359447b1a917c79091fa4232 (127.19.34.66:34735), 0385489d421847c287073da102281bde (127.19.34.65:43215)
I20260619 14:52:00.960733 24678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:00.959227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:34735 (local address 127.19.34.67:51843)
0619 14:52:00.959363 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:00.959394 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:00.959483 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:00.959760 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:00.959763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:00.959774 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:00.960262 (+ 488us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:52:00.960265 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:00.960277 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:52:00.960324 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:00.960521 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:00.960524 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:00.960527 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:52:00.960592 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 14:52:00.960623 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 14:52:00.960733 24685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:00.959337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:51843 (local address 127.19.34.66:34735)
0619 14:52:00.959468 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:00.959472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:00.959487 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:52:00.959561 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:00.959566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:00.959605 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:00.959695 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:00.960410 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:00.960413 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:00.960415 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:00.960425 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:00.960447 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:00.960454 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:00.960535 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:00.960589 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 14:52:00.960623 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:52:00.960978 24699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:00.959488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:43215 (local address 127.19.34.67:34245)
0619 14:52:00.959776 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:00.959787 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:00.959870 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:00.960088 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:00.960089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:00.960100 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:00.960490 (+ 390us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:52:00.960492 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:00.960501 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:52:00.960530 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:00.960752 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:00.960754 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:00.960757 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:52:00.960831 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 14:52:00.960863 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":124,"threads_started":1}
I20260619 14:52:00.961028 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:00.959598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:34245 (local address 127.19.34.65:43215)
0619 14:52:00.959741 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:00.959747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:00.959866 (+ 119us) server_negotiation.cc:407] Connection header received
0619 14:52:00.959956 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:00.959961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:00.959996 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:00.960070 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:00.960631 (+ 561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:00.960634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:00.960636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:00.960646 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:00.960664 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:00.960677 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:00.960758 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:00.960881 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 14:52:00.960913 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:52:00.961411 24499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4e8a4ed5842d39cea60fc23c271ad" candidate_uuid: "ad8d4b19165f44ea99a8039371c6228f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb61a0a2359447b1a917c79091fa4232" is_pre_election: true
I20260619 14:52:00.961514 24368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4e8a4ed5842d39cea60fc23c271ad" candidate_uuid: "ad8d4b19165f44ea99a8039371c6228f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0385489d421847c287073da102281bde" is_pre_election: true
I20260619 14:52:00.961658 24499 raft_consensus.cc:2454] T c9e4e8a4ed5842d39cea60fc23c271ad P eb61a0a2359447b1a917c79091fa4232 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad8d4b19165f44ea99a8039371c6228f in term 0.
I20260619 14:52:00.961800 24368 raft_consensus.cc:2454] T c9e4e8a4ed5842d39cea60fc23c271ad P 0385489d421847c287073da102281bde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad8d4b19165f44ea99a8039371c6228f in term 0.
I20260619 14:52:00.962045 24566 leader_election.cc:302] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad8d4b19165f44ea99a8039371c6228f, eb61a0a2359447b1a917c79091fa4232; no voters:
I20260619 14:52:00.962265 24696 raft_consensus.cc:2781] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:52:00.962350 24696 raft_consensus.cc:482] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:52:00.962401 24696 raft_consensus.cc:3037] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:00.963338 24696 raft_consensus.cc:504] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb61a0a2359447b1a917c79091fa4232" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34735 } } peers { permanent_uuid: "0385489d421847c287073da102281bde" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 43215 } } peers { permanent_uuid: "ad8d4b19165f44ea99a8039371c6228f" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 34957 } }
I20260619 14:52:00.963618 24696 leader_election.cc:288] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f [CANDIDATE]: Term 1 election: Requested vote from peers eb61a0a2359447b1a917c79091fa4232 (127.19.34.66:34735), 0385489d421847c287073da102281bde (127.19.34.65:43215)
I20260619 14:52:00.963872 24499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4e8a4ed5842d39cea60fc23c271ad" candidate_uuid: "ad8d4b19165f44ea99a8039371c6228f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb61a0a2359447b1a917c79091fa4232"
I20260619 14:52:00.963999 24368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4e8a4ed5842d39cea60fc23c271ad" candidate_uuid: "ad8d4b19165f44ea99a8039371c6228f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0385489d421847c287073da102281bde"
I20260619 14:52:00.964051 24499 raft_consensus.cc:3037] T c9e4e8a4ed5842d39cea60fc23c271ad P eb61a0a2359447b1a917c79091fa4232 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:00.964097 24368 raft_consensus.cc:3037] T c9e4e8a4ed5842d39cea60fc23c271ad P 0385489d421847c287073da102281bde [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:00.965317 24368 raft_consensus.cc:2454] T c9e4e8a4ed5842d39cea60fc23c271ad P 0385489d421847c287073da102281bde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad8d4b19165f44ea99a8039371c6228f in term 1.
I20260619 14:52:00.965317 24499 raft_consensus.cc:2454] T c9e4e8a4ed5842d39cea60fc23c271ad P eb61a0a2359447b1a917c79091fa4232 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad8d4b19165f44ea99a8039371c6228f in term 1.
I20260619 14:52:00.965583 24566 leader_election.cc:302] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad8d4b19165f44ea99a8039371c6228f, eb61a0a2359447b1a917c79091fa4232; no voters:
I20260619 14:52:00.965708 24696 raft_consensus.cc:2781] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:00.965903 24696 raft_consensus.cc:686] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f [term 1 LEADER]: Becoming Leader. State: Replica: ad8d4b19165f44ea99a8039371c6228f, State: Running, Role: LEADER
I20260619 14:52:00.966187 24696 consensus_queue.cc:237] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb61a0a2359447b1a917c79091fa4232" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34735 } } peers { permanent_uuid: "0385489d421847c287073da102281bde" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 43215 } } peers { permanent_uuid: "ad8d4b19165f44ea99a8039371c6228f" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 34957 } }
I20260619 14:52:00.969091 24224 catalog_manager.cc:5495] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f reported cstate change: term changed from 0 to 1, leader changed from <none> to ad8d4b19165f44ea99a8039371c6228f (127.19.34.67). New cstate: current_term: 1 leader_uuid: "ad8d4b19165f44ea99a8039371c6228f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb61a0a2359447b1a917c79091fa4232" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0385489d421847c287073da102281bde" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad8d4b19165f44ea99a8039371c6228f" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 34957 } health_report { overall_health: HEALTHY } } }
I20260619 14:52:01.027494 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:01.026012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49292 (local address 127.19.34.67:34957)
0619 14:52:01.026165 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:01.026168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:01.026181 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:52:01.026252 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:01.026254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:01.026309 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:01.026379 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:01.027068 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:01.027070 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:01.027072 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:01.027080 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:01.027095 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:01.027102 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:01.027193 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:01.027255 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 14:52:01.027332 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:52:01.033095 24499 raft_consensus.cc:1261] T c9e4e8a4ed5842d39cea60fc23c271ad P eb61a0a2359447b1a917c79091fa4232 [term 1 FOLLOWER]: Refusing update from remote peer ad8d4b19165f44ea99a8039371c6228f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:01.033095 24368 raft_consensus.cc:1261] T c9e4e8a4ed5842d39cea60fc23c271ad P 0385489d421847c287073da102281bde [term 1 FOLLOWER]: Refusing update from remote peer ad8d4b19165f44ea99a8039371c6228f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:01.033809 24696 consensus_queue.cc:1035] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0385489d421847c287073da102281bde" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:01.033991 24700 consensus_queue.cc:1035] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb61a0a2359447b1a917c79091fa4232" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:01.042629 24705 mvcc.cc:205] Tried to move back new op lower bound from 7298583433344708608 to 7298583433079640064. Current Snapshot: MvccSnapshot[applied={T|T < 7298583433344708608}]
I20260619 14:52:01.045135 24704 mvcc.cc:205] Tried to move back new op lower bound from 7298583433344708608 to 7298583433079640064. Current Snapshot: MvccSnapshot[applied={T|T < 7298583433344708608 or (T in {7298583433344708608})}]
I20260619 14:52:01.048938 24226 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:01.049111 24226 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:01.051249 24226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e6f86d9cc2d4b8d88cee1825caed930: Sending DeleteTablet for 3 replicas of tablet c9e4e8a4ed5842d39cea60fc23c271ad
I20260619 14:52:01.051913 24610 tablet_service.cc:1514] Processing DeleteTablet for tablet c9e4e8a4ed5842d39cea60fc23c271ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:01 UTC) from {username='slave'} at 127.0.0.1:49280
I20260619 14:52:01.051995 24479 tablet_service.cc:1514] Processing DeleteTablet for tablet c9e4e8a4ed5842d39cea60fc23c271ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:01 UTC) from {username='slave'} at 127.0.0.1:50680
I20260619 14:52:01.052134 24348 tablet_service.cc:1514] Processing DeleteTablet for tablet c9e4e8a4ed5842d39cea60fc23c271ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:01 UTC) from {username='slave'} at 127.0.0.1:50120
I20260619 14:52:01.052510 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 24286
I20260619 14:52:01.056485 24714 tablet_replica.cc:330] T c9e4e8a4ed5842d39cea60fc23c271ad P eb61a0a2359447b1a917c79091fa4232: stopping tablet replica
I20260619 14:52:01.056746 24714 raft_consensus.cc:2229] T c9e4e8a4ed5842d39cea60fc23c271ad P eb61a0a2359447b1a917c79091fa4232 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:01.056995 24714 raft_consensus.cc:2258] T c9e4e8a4ed5842d39cea60fc23c271ad P eb61a0a2359447b1a917c79091fa4232 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:01.057287 24713 tablet_replica.cc:330] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f: stopping tablet replica
I20260619 14:52:01.057528 24713 raft_consensus.cc:2229] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:52:01.057822 24713 raft_consensus.cc:2258] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:01.058480 24714 ts_tablet_manager.cc:1905] T c9e4e8a4ed5842d39cea60fc23c271ad P eb61a0a2359447b1a917c79091fa4232: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:01.058975 24713 ts_tablet_manager.cc:1905] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 14:52:01.059366 24213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.34.65:43215 (error 108)
I20260619 14:52:01.059999 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 24417
I20260619 14:52:01.061736 24713 ts_tablet_manager.cc:1918] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:01.061841 24713 log.cc:1196] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/c9e4e8a4ed5842d39cea60fc23c271ad
I20260619 14:52:01.062093 24713 ts_tablet_manager.cc:1939] T c9e4e8a4ed5842d39cea60fc23c271ad P ad8d4b19165f44ea99a8039371c6228f: Deleting consensus metadata
I20260619 14:52:01.063020 24212 catalog_manager.cc:4841] TS ad8d4b19165f44ea99a8039371c6228f (127.19.34.67:34957): tablet c9e4e8a4ed5842d39cea60fc23c271ad (table test-table [id=d9cf62d9b3364addba4e582f404e49d6]) successfully deleted
I20260619 14:52:01.063850 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:01.063516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:43215 (local address 127.0.0.1:50136)
0619 14:52:01.063691 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:01.063760 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.65:43215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260619 14:52:01.064073 24213 catalog_manager.cc:4584] TS 0385489d421847c287073da102281bde (127.19.34.65:43215): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c9e4e8a4ed5842d39cea60fc23c271ad: Network error: Client connection negotiation failed: client connection to 127.19.34.65:43215: connect: Connection refused (error 111)
I20260619 14:52:01.066429 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 24548
I20260619 14:52:01.066555 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:01.066332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:34735 (local address 127.0.0.1:50686)
0619 14:52:01.066429 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:01.066486 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.66:34735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 14:52:01.071749 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 24195
2026-06-19T14:52:01Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1028 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 14:52:01.091741 19593 mini_oidc.cc:152] Starting JWKS server
I20260619 14:52:01.093161 19593 webserver.cc:460] Webserver started at https://127.0.0.1:36627/ using document root <none> and password file <none>
I20260619 14:52:01.093328 19593 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 14:52:01.093575 19593 webserver.cc:460] Webserver started at http://127.0.0.1:37989/ using document root <none> and password file <none>
2026-06-19T14:52:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:52:01Z Disabled control of system clock
I20260619 14:52:01.101286 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:41801
--webserver_interface=127.19.34.126
--webserver_port=0
--builtin_ntp_servers=127.19.34.84:36001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:41801
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36627/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880697016847-19593-0/testchainca.cert with env {}
W20260619 14:52:01.195708 24722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:01.195899 24722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:01.195930 24722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:01.199149 24722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:01.199205 24722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:01.199219 24722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:01.199232 24722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:01.199245 24722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:52:01.199258 24722 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:52:01.199270 24722 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36627/jwks.json
I20260619 14:52:01.203454 24722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:36001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:41801
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:41801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36627/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880697016847-19593-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:01.204146 24722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:01.204985 24722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:01.211777 24728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:01.211833 24722 server_base.cc:1034] running on GCE node
W20260619 14:52:01.211838 24727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:01.211997 24730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:01.212502 24722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:01.213330 24722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:01.214514 24722 hybrid_clock.cc:648] HybridClock initialized: now 1781880721214497 us; error 26 us; skew 500 ppm
I20260619 14:52:01.216123 24722 webserver.cc:460] Webserver started at http://127.19.34.126:42477/ using document root <none> and password file <none>
I20260619 14:52:01.216619 24722 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:01.216704 24722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:01.216884 24722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:01.218417 24722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "383a3b61f6e44a74997ce7737de04737"
format_stamp: "Formatted at 2026-06-19 14:52:01 on dist-test-slave-3qdh"
server_key: "188e96f77609c609b5d02d1f0ce24e99"
server_key_iv: "cfc933e7429cbe2073b0f2dc3c67c6cc"
server_key_version: "encryptionkey@0"
I20260619 14:52:01.218879 24722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "383a3b61f6e44a74997ce7737de04737"
format_stamp: "Formatted at 2026-06-19 14:52:01 on dist-test-slave-3qdh"
server_key: "188e96f77609c609b5d02d1f0ce24e99"
server_key_iv: "cfc933e7429cbe2073b0f2dc3c67c6cc"
server_key_version: "encryptionkey@0"
I20260619 14:52:01.221807 24722 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:52:01.223681 24736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:01.224732 24722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:01.224858 24722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "383a3b61f6e44a74997ce7737de04737"
format_stamp: "Formatted at 2026-06-19 14:52:01 on dist-test-slave-3qdh"
server_key: "188e96f77609c609b5d02d1f0ce24e99"
server_key_iv: "cfc933e7429cbe2073b0f2dc3c67c6cc"
server_key_version: "encryptionkey@0"
I20260619 14:52:01.224958 24722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:01.242496 24722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:01.242971 24722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:01.243124 24722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:01.249228 24722 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:41801
I20260619 14:52:01.249254 24788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:41801 every 8 connection(s)
I20260619 14:52:01.249949 24722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:52:01.252602 24789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:01.255390 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 24722
I20260619 14:52:01.255524 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:52:01.255781 19593 external_mini_cluster.cc:1428] Setting key 32a4bcdd5c23ec239ffa073526c864b3
I20260619 14:52:01.257627 24789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737: Bootstrap starting.
I20260619 14:52:01.259760 24789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:01.260612 24789 log.cc:824] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:01.262391 24789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737: No bootstrap required, opened a new log
I20260619 14:52:01.263801 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:01.256927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39516 (local address 127.19.34.126:41801)
0619 14:52:01.257433 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:01.257447 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:01.257476 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:52:01.258023 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:01.258038 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:01.258444 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:01.258689 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:01.258983 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:01.259926 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:01.261125 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:01.261605 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:01.261711 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:01.262365 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:01.262369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:01.262378 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:01.262738 (+ 360us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:01.262782 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:01.262799 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:01.262864 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:01.263009 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 14:52:01.263254 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":112,"threads_started":1}
I20260619 14:52:01.265641 24789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383a3b61f6e44a74997ce7737de04737" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41801 } }
I20260619 14:52:01.265826 24789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:01.265913 24789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 383a3b61f6e44a74997ce7737de04737, State: Initialized, Role: FOLLOWER
I20260619 14:52:01.266182 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:52:01.266332 24789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383a3b61f6e44a74997ce7737de04737" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41801 } }
I20260619 14:52:01.266449 24789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:52:01.266496 24789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:52:01.266568 24789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:01.267331 24789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383a3b61f6e44a74997ce7737de04737" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41801 } }
I20260619 14:52:01.267606 24789 leader_election.cc:302] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 383a3b61f6e44a74997ce7737de04737; no voters:
I20260619 14:52:01.267848 24789 leader_election.cc:288] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:52:01.267959 24794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:01.268208 24794 raft_consensus.cc:686] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737 [term 1 LEADER]: Becoming Leader. State: Replica: 383a3b61f6e44a74997ce7737de04737, State: Running, Role: LEADER
I20260619 14:52:01.268513 24794 consensus_queue.cc:237] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383a3b61f6e44a74997ce7737de04737" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41801 } }
I20260619 14:52:01.268891 24789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:01.269939 24795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "383a3b61f6e44a74997ce7737de04737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383a3b61f6e44a74997ce7737de04737" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41801 } } }
I20260619 14:52:01.269971 24796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 383a3b61f6e44a74997ce7737de04737. Latest consensus state: current_term: 1 leader_uuid: "383a3b61f6e44a74997ce7737de04737" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383a3b61f6e44a74997ce7737de04737" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41801 } } }
I20260619 14:52:01.270032 24795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:01.270061 24796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:01.270421 24803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:52:01.273304 24803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:52:01.278175 24803 catalog_manager.cc:1347] Generated new cluster ID: 0bfbcf91977048d6a330a5d764885562
I20260619 14:52:01.278245 24803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:52:01.285287 24803 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:52:01.285993 24803 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:52:01.294438 24803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 383a3b61f6e44a74997ce7737de04737: Generated new TSK 0
I20260619 14:52:01.294982 24803 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 14:52:01.322160 24819 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 14:52:01.322283 24819 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 14:52:01.322788 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:01.306930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39532 (local address 127.19.34.126:41801)
0619 14:52:01.307047 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:01.307051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:01.307241 (+ 190us) server_negotiation.cc:407] Connection header received
0619 14:52:01.307441 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:01.307444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:01.307489 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:01.307582 (+ 93us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:52:01.307817 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:01.308389 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:01.309228 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:01.309443 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:01.309547 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:01.309851 (+ 304us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:52:01.322425 (+ 12574us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36627/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 14:52:01.322643 (+ 218us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39532: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36627/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 14:52:01.322863 24792 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39532: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36627/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 14:52:01.323805 24818 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:01.306834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:41801 (local address 127.0.0.1:39532)
0619 14:52:01.307161 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:01.307172 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:01.307300 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:01.307601 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:01.307604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:01.307611 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:52:01.307720 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:01.307726 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:01.308499 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:01.308502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:01.309092 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:01.309099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:01.309551 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:01.309554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:01.309643 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:01.309688 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:52:01.322605 (+ 12917us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36627/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 14:52:01.322802 (+ 197us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.34.126:41801: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36627/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":111,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.19.34.126:41801: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36627/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 14:52:01.324522 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 24722
2026-06-19T14:52:01Z chronyd exiting
I20260619 14:52:01.360953 19593 test_util.cc:175] -----------------------------------------------
I20260619 14:52:01.361006 19593 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880697016847-19593-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (270 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 14:52:01.361860 19593 mini_oidc.cc:152] Starting JWKS server
I20260619 14:52:01.362556 19593 webserver.cc:460] Webserver started at https://127.0.0.1:44605/ using document root <none> and password file <none>
I20260619 14:52:01.362656 19593 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 14:52:01.362917 19593 webserver.cc:460] Webserver started at http://127.0.0.1:42109/ using document root <none> and password file <none>
2026-06-19T14:52:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:52:01Z Disabled control of system clock
I20260619 14:52:01.370321 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:38713
--webserver_interface=127.19.34.126
--webserver_port=0
--builtin_ntp_servers=127.19.34.84:40053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:38713
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44605/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880697016847-19593-0/testchainca.cert with env {}
W20260619 14:52:01.463778 24825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:01.464003 24825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:01.464064 24825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:01.467775 24825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:01.467860 24825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:01.467902 24825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:01.467942 24825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:01.467971 24825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:52:01.468016 24825 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:52:01.468048 24825 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44605/jwks.json
I20260619 14:52:01.472455 24825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:40053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:38713
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:38713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44605/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880697016847-19593-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:01.473227 24825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:01.474184 24825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:01.481186 24833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:01.481194 24830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:01.481194 24831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:01.481377 24825 server_base.cc:1034] running on GCE node
I20260619 14:52:01.481920 24825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:01.482702 24825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:01.483839 24825 hybrid_clock.cc:648] HybridClock initialized: now 1781880721483815 us; error 36 us; skew 500 ppm
I20260619 14:52:01.485471 24825 webserver.cc:460] Webserver started at http://127.19.34.126:40777/ using document root <none> and password file <none>
I20260619 14:52:01.485940 24825 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:01.485991 24825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:01.486172 24825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:01.487721 24825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "65d08c3fe7af4bbc94c0817c909e8087"
format_stamp: "Formatted at 2026-06-19 14:52:01 on dist-test-slave-3qdh"
server_key: "224d3b05ca1dc8efd2217d6614e48715"
server_key_iv: "7a155d66e32b2775a346b841556b9ab9"
server_key_version: "encryptionkey@0"
I20260619 14:52:01.488212 24825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "65d08c3fe7af4bbc94c0817c909e8087"
format_stamp: "Formatted at 2026-06-19 14:52:01 on dist-test-slave-3qdh"
server_key: "224d3b05ca1dc8efd2217d6614e48715"
server_key_iv: "7a155d66e32b2775a346b841556b9ab9"
server_key_version: "encryptionkey@0"
I20260619 14:52:01.491130 24825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 14:52:01.493072 24839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:01.494037 24825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:01.494153 24825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "65d08c3fe7af4bbc94c0817c909e8087"
format_stamp: "Formatted at 2026-06-19 14:52:01 on dist-test-slave-3qdh"
server_key: "224d3b05ca1dc8efd2217d6614e48715"
server_key_iv: "7a155d66e32b2775a346b841556b9ab9"
server_key_version: "encryptionkey@0"
I20260619 14:52:01.494243 24825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:01.510295 24825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:01.510771 24825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:01.510926 24825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:01.516801 24825 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:38713
I20260619 14:52:01.516824 24891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:38713 every 8 connection(s)
I20260619 14:52:01.517490 24825 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:52:01.520174 24892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:01.524639 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 24825
I20260619 14:52:01.524768 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:52:01.524765 24892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087: Bootstrap starting.
I20260619 14:52:01.525020 19593 external_mini_cluster.cc:1428] Setting key 0867112fe037e2c5f80b574c3ecead3f
I20260619 14:52:01.526961 24892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:01.527606 24892 log.cc:824] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:01.529277 24892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087: No bootstrap required, opened a new log
I20260619 14:52:01.532891 24895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:01.526101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34292 (local address 127.19.34.126:38713)
0619 14:52:01.526684 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:01.526704 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:01.526736 (+ 32us) server_negotiation.cc:407] Connection header received
0619 14:52:01.527330 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:01.527346 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:01.527708 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:01.528020 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:01.528405 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:01.529230 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:01.530299 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:01.530687 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:01.530784 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:01.531481 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:01.531485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:01.531495 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:01.531798 (+ 303us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:01.531841 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:01.531857 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:01.531918 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:01.532077 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 14:52:01.532302 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":110,"threads_started":1}
I20260619 14:52:01.532802 24892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d08c3fe7af4bbc94c0817c909e8087" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38713 } }
I20260619 14:52:01.533025 24892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:01.533075 24892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d08c3fe7af4bbc94c0817c909e8087, State: Initialized, Role: FOLLOWER
I20260619 14:52:01.533389 24892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d08c3fe7af4bbc94c0817c909e8087" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38713 } }
I20260619 14:52:01.533475 24892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:52:01.533507 24892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:52:01.533555 24892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:01.534364 24892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d08c3fe7af4bbc94c0817c909e8087" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38713 } }
I20260619 14:52:01.534765 24892 leader_election.cc:302] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65d08c3fe7af4bbc94c0817c909e8087; no voters:
I20260619 14:52:01.535081 24892 leader_election.cc:288] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:52:01.535167 24897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:01.535403 24897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087 [term 1 LEADER]: Becoming Leader. State: Replica: 65d08c3fe7af4bbc94c0817c909e8087, State: Running, Role: LEADER
I20260619 14:52:01.535480 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:52:01.535705 24897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d08c3fe7af4bbc94c0817c909e8087" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38713 } }
I20260619 14:52:01.536079 24892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:01.536775 24899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65d08c3fe7af4bbc94c0817c909e8087. Latest consensus state: current_term: 1 leader_uuid: "65d08c3fe7af4bbc94c0817c909e8087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d08c3fe7af4bbc94c0817c909e8087" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38713 } } }
I20260619 14:52:01.536890 24899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:01.537066 24898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65d08c3fe7af4bbc94c0817c909e8087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65d08c3fe7af4bbc94c0817c909e8087" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38713 } } }
I20260619 14:52:01.537138 24898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:01.537369 24906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:52:01.539831 24906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:52:01.543949 24906 catalog_manager.cc:1347] Generated new cluster ID: 6df4d0a54a824212963604889edbedb6
I20260619 14:52:01.544001 24906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:52:01.559156 24906 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:52:01.559810 24906 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:52:01.571553 24906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65d08c3fe7af4bbc94c0817c909e8087: Generated new TSK 0
I20260619 14:52:01.572052 24906 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 14:52:01.591895 24922 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 14:52:01.591952 24922 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 14:52:01.592474 24895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:01.579015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34294 (local address 127.19.34.126:38713)
0619 14:52:01.579127 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:01.579131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:01.579315 (+ 184us) server_negotiation.cc:407] Connection header received
0619 14:52:01.579447 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:01.579449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:01.579488 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:01.579553 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:52:01.579764 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:01.580205 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:01.581255 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:01.581431 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:01.581495 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:01.581754 (+ 259us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:52:01.592164 (+ 10410us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 14:52:01.592382 (+ 218us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34294: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 14:52:01.592532 24895 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34294: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 14:52:01.592772 24921 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:01.578971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:38713 (local address 127.0.0.1:34294)
0619 14:52:01.579236 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:01.579249 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:01.579344 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:01.579580 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:01.579582 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:01.579589 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:52:01.579679 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:01.579685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:01.580322 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:01.580325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:01.581151 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:01.581160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:01.581525 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:01.581529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:01.581608 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:01.581631 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:52:01.592311 (+ 10680us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 14:52:01.592444 (+ 133us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.34.126:38713: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":92,"threads_started":1}
I20260619 14:52:01.593277 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 24825
2026-06-19T14:52:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (269 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 14:52:01.631659 19593 mini_oidc.cc:152] Starting JWKS server
I20260619 14:52:01.632431 19593 webserver.cc:460] Webserver started at https://127.0.0.1:39941/ using document root <none> and password file <none>
I20260619 14:52:01.632532 19593 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 14:52:01.632791 19593 webserver.cc:460] Webserver started at http://127.0.0.1:32899/ using document root <none> and password file <none>
2026-06-19T14:52:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:52:01Z Disabled control of system clock
I20260619 14:52:01.640585 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:41941
--webserver_interface=127.19.34.126
--webserver_port=0
--builtin_ntp_servers=127.19.34.84:42933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:41941
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39941/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 14:52:01.736616 24928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:01.736819 24928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:01.736851 24928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:01.740149 24928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:01.740190 24928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:01.740204 24928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:01.740218 24928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:01.740232 24928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:52:01.740245 24928 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:52:01.740257 24928 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39941/jwks.json
I20260619 14:52:01.744541 24928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:42933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:41941
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:41941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39941/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:01.745373 24928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:01.746538 24928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:01.754091 24933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:01.754148 24934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:01.754106 24936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:01.754395 24928 server_base.cc:1034] running on GCE node
I20260619 14:52:01.754930 24928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:01.755888 24928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:01.757058 24928 hybrid_clock.cc:648] HybridClock initialized: now 1781880721757050 us; error 28 us; skew 500 ppm
I20260619 14:52:01.758756 24928 webserver.cc:460] Webserver started at http://127.19.34.126:35025/ using document root <none> and password file <none>
I20260619 14:52:01.759298 24928 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:01.759383 24928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:01.759580 24928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:01.761394 24928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "547510950ceb416dbf551efff46c475f"
format_stamp: "Formatted at 2026-06-19 14:52:01 on dist-test-slave-3qdh"
server_key: "9eaa9bbdf609a9a01d71ab03bf1932a3"
server_key_iv: "a3bcb435f36c871a572474414b7b2e01"
server_key_version: "encryptionkey@0"
I20260619 14:52:01.761890 24928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "547510950ceb416dbf551efff46c475f"
format_stamp: "Formatted at 2026-06-19 14:52:01 on dist-test-slave-3qdh"
server_key: "9eaa9bbdf609a9a01d71ab03bf1932a3"
server_key_iv: "a3bcb435f36c871a572474414b7b2e01"
server_key_version: "encryptionkey@0"
I20260619 14:52:01.764986 24928 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:52:01.766973 24942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:01.767967 24928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:01.768090 24928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "547510950ceb416dbf551efff46c475f"
format_stamp: "Formatted at 2026-06-19 14:52:01 on dist-test-slave-3qdh"
server_key: "9eaa9bbdf609a9a01d71ab03bf1932a3"
server_key_iv: "a3bcb435f36c871a572474414b7b2e01"
server_key_version: "encryptionkey@0"
I20260619 14:52:01.768220 24928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:01.792096 24928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:01.792671 24928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:01.792841 24928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:01.799023 24928 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:41941
I20260619 14:52:01.799031 24994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:41941 every 8 connection(s)
I20260619 14:52:01.799793 24928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:52:01.802423 24995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:01.804986 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 24928
I20260619 14:52:01.805073 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:52:01.805331 19593 external_mini_cluster.cc:1428] Setting key b480b197dc23838a375b812995331889
I20260619 14:52:01.807384 24995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f: Bootstrap starting.
I20260619 14:52:01.809422 24995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:01.810113 24995 log.cc:824] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:01.811986 24995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f: No bootstrap required, opened a new log
I20260619 14:52:01.813272 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:01.806521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.19.34.126:41941)
0619 14:52:01.807018 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:01.807031 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:01.807059 (+ 28us) server_negotiation.cc:407] Connection header received
0619 14:52:01.807605 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:01.807620 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:01.807954 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:01.808179 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:01.808561 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:01.809424 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:01.810298 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:01.810772 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:01.810932 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:01.811769 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:01.811773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:01.811784 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:01.812155 (+ 371us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:01.812205 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:01.812223 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:01.812331 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:01.812477 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 14:52:01.812697 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":107,"threads_started":1}
I20260619 14:52:01.815732 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:52:01.815646 24995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547510950ceb416dbf551efff46c475f" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41941 } }
I20260619 14:52:01.815879 24995 raft_consensus.cc:374] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:01.815953 24995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 547510950ceb416dbf551efff46c475f, State: Initialized, Role: FOLLOWER
I20260619 14:52:01.816334 24995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547510950ceb416dbf551efff46c475f" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41941 } }
I20260619 14:52:01.816483 24995 raft_consensus.cc:388] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:52:01.816574 24995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:52:01.816664 24995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:01.817507 24995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547510950ceb416dbf551efff46c475f" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41941 } }
I20260619 14:52:01.817744 24995 leader_election.cc:302] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 547510950ceb416dbf551efff46c475f; no voters:
I20260619 14:52:01.817931 24995 leader_election.cc:288] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:52:01.818030 25000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:01.818233 25000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f [term 1 LEADER]: Becoming Leader. State: Replica: 547510950ceb416dbf551efff46c475f, State: Running, Role: LEADER
I20260619 14:52:01.818517 25000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547510950ceb416dbf551efff46c475f" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41941 } }
I20260619 14:52:01.818928 24995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:01.820001 25001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "547510950ceb416dbf551efff46c475f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547510950ceb416dbf551efff46c475f" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41941 } } }
I20260619 14:52:01.820006 25002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 547510950ceb416dbf551efff46c475f. Latest consensus state: current_term: 1 leader_uuid: "547510950ceb416dbf551efff46c475f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "547510950ceb416dbf551efff46c475f" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41941 } } }
I20260619 14:52:01.820109 25001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:01.820113 25002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:01.820453 25009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:52:01.822676 25009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:52:01.827430 25009 catalog_manager.cc:1347] Generated new cluster ID: fae803c91f174ebb80c7bbad72b98ced
I20260619 14:52:01.827517 25009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:52:01.866170 25009 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:52:01.867257 25009 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:52:01.871501 25009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 547510950ceb416dbf551efff46c475f: Generated new TSK 0
I20260619 14:52:01.872191 25009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:52:01.888586 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:01.884646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60520 (local address 127.19.34.126:41941)
0619 14:52:01.884745 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:01.884748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:01.884761 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:52:01.884910 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:01.884912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:01.884952 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:01.885033 (+ 81us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:52:01.885284 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:01.885770 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:01.886820 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:01.886998 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:01.887061 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:01.887324 (+ 263us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:52:01.888236 (+ 912us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39941/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 14:52:01.888496 (+ 260us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60520: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39941/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":27}
W20260619 14:52:01.888650 24998 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60520: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39941/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 14:52:01.888731 25024 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:01.884364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:41941 (local address 127.0.0.1:60520)
0619 14:52:01.884691 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:01.884706 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:01.884804 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:01.885053 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:01.885057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:01.885064 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:52:01.885183 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:01.885189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:01.885877 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:01.885880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:01.886722 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:01.886731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:01.887092 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:01.887096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:01.887172 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:01.887195 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:52:01.888446 (+ 1251us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39941/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 14:52:01.888573 (+ 127us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.34.126:41941: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39941/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":114,"threads_started":1}
I20260619 14:52:01.889163 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 24928
E20260619 14:52:01.901118 25025 webserver.cc:533] Webserver: sslize failed:
E20260619 14:52:01.901168 25025 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T14:52:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (296 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T14:52:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:52:01Z Disabled control of system clock
I20260619 14:52:01.935969 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:35581
--webserver_interface=127.19.34.126
--webserver_port=0
--builtin_ntp_servers=127.19.34.84:33055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:35581
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880697016847-19593-0/testchainca.cert with env {}
W20260619 14:52:02.033865 25029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:02.034075 25029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:02.034107 25029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:02.038235 25029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:02.038277 25029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:02.038293 25029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:02.038308 25029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:02.038322 25029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:52:02.038336 25029 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:52:02.038355 25029 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 14:52:02.042851 25029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:33055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:35581
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:35581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880697016847-19593-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:02.043569 25029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:02.044436 25029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:02.051611 25034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:02.051736 25029 server_base.cc:1034] running on GCE node
W20260619 14:52:02.051640 25035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:02.051643 25037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:02.052423 25029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:02.053367 25029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:02.054546 25029 hybrid_clock.cc:648] HybridClock initialized: now 1781880722054531 us; error 25 us; skew 500 ppm
I20260619 14:52:02.056156 25029 webserver.cc:460] Webserver started at http://127.19.34.126:40217/ using document root <none> and password file <none>
I20260619 14:52:02.056674 25029 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:02.056777 25029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:02.056991 25029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:02.058753 25029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "29139db5da9f44e28c1e1fdd64d3bcf6"
format_stamp: "Formatted at 2026-06-19 14:52:02 on dist-test-slave-3qdh"
server_key: "903c576b523cb32df3d4b4571b87ad8f"
server_key_iv: "5e6f5356fa29d99d03990dcc078a7d3f"
server_key_version: "encryptionkey@0"
I20260619 14:52:02.059180 25029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "29139db5da9f44e28c1e1fdd64d3bcf6"
format_stamp: "Formatted at 2026-06-19 14:52:02 on dist-test-slave-3qdh"
server_key: "903c576b523cb32df3d4b4571b87ad8f"
server_key_iv: "5e6f5356fa29d99d03990dcc078a7d3f"
server_key_version: "encryptionkey@0"
I20260619 14:52:02.062281 25029 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:52:02.064463 25043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:02.065562 25029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:02.065702 25029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "29139db5da9f44e28c1e1fdd64d3bcf6"
format_stamp: "Formatted at 2026-06-19 14:52:02 on dist-test-slave-3qdh"
server_key: "903c576b523cb32df3d4b4571b87ad8f"
server_key_iv: "5e6f5356fa29d99d03990dcc078a7d3f"
server_key_version: "encryptionkey@0"
I20260619 14:52:02.065832 25029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:02.088076 25029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:02.088645 25029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:02.088814 25029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:02.094581 25029 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:35581
I20260619 14:52:02.094591 25095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:35581 every 8 connection(s)
I20260619 14:52:02.095373 25029 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:52:02.098045 25096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:02.100543 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 25029
I20260619 14:52:02.100665 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:52:02.100948 19593 external_mini_cluster.cc:1428] Setting key ba167d4178169907d9fe9e7d31ad87a5
I20260619 14:52:02.103467 25096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6: Bootstrap starting.
I20260619 14:52:02.105506 25096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:02.106202 25096 log.cc:824] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:02.107926 25096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6: No bootstrap required, opened a new log
I20260619 14:52:02.109138 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:02.102154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51924 (local address 127.19.34.126:35581)
0619 14:52:02.102682 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:02.102695 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:02.102725 (+ 30us) server_negotiation.cc:407] Connection header received
0619 14:52:02.103287 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:02.103302 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:02.103695 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:02.103992 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:02.104488 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:02.105410 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:02.106309 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:02.106755 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:02.106911 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:02.107666 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:02.107670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:02.107680 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:02.108031 (+ 351us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:02.108093 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:02.108112 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:02.108208 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:02.108358 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 14:52:02.108572 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":99,"threads_started":1}
I20260619 14:52:02.111327 25096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29139db5da9f44e28c1e1fdd64d3bcf6" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 35581 } }
I20260619 14:52:02.111521 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:52:02.111547 25096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:02.111601 25096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29139db5da9f44e28c1e1fdd64d3bcf6, State: Initialized, Role: FOLLOWER
I20260619 14:52:02.111922 25096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29139db5da9f44e28c1e1fdd64d3bcf6" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 35581 } }
I20260619 14:52:02.112046 25096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:52:02.112126 25096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:52:02.112213 25096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:02.113093 25096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29139db5da9f44e28c1e1fdd64d3bcf6" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 35581 } }
I20260619 14:52:02.113390 25096 leader_election.cc:302] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29139db5da9f44e28c1e1fdd64d3bcf6; no voters:
I20260619 14:52:02.113646 25096 leader_election.cc:288] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:52:02.113744 25101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:02.113916 25101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6 [term 1 LEADER]: Becoming Leader. State: Replica: 29139db5da9f44e28c1e1fdd64d3bcf6, State: Running, Role: LEADER
I20260619 14:52:02.114180 25101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29139db5da9f44e28c1e1fdd64d3bcf6" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 35581 } }
I20260619 14:52:02.114667 25096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:02.115494 25103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29139db5da9f44e28c1e1fdd64d3bcf6. Latest consensus state: current_term: 1 leader_uuid: "29139db5da9f44e28c1e1fdd64d3bcf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29139db5da9f44e28c1e1fdd64d3bcf6" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 35581 } } }
I20260619 14:52:02.115801 25102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29139db5da9f44e28c1e1fdd64d3bcf6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29139db5da9f44e28c1e1fdd64d3bcf6" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 35581 } } }
I20260619 14:52:02.115911 25103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:02.115942 25102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:02.116590 25110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:52:02.118937 25110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:52:02.123716 25110 catalog_manager.cc:1347] Generated new cluster ID: 7dd11063f3fa4ee7ae5d0294c5a24ea1
I20260619 14:52:02.123769 25110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:52:02.138940 25110 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:52:02.139873 25110 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:52:02.151718 25110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29139db5da9f44e28c1e1fdd64d3bcf6: Generated new TSK 0
I20260619 14:52:02.152366 25110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:52:02.605860 25099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:02.168994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.19.34.126:35581)
0619 14:52:02.169106 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:02.169111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:02.169282 (+ 171us) server_negotiation.cc:407] Connection header received
0619 14:52:02.169393 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:02.169396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:02.169446 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:02.169548 (+ 102us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:52:02.169708 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:02.170425 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:02.171171 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:02.171469 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:02.171559 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:02.171778 (+ 219us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:52:02.605480 (+433702us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 14:52:02.605745 (+ 265us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51936: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 14:52:02.605966 25099 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51936: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 14:52:02.606063 25125 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:02.168922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:35581 (local address 127.0.0.1:51936)
0619 14:52:02.169203 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:02.169215 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:02.169303 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:02.169550 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:02.169552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:02.169557 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:52:02.169629 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:02.169633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:02.170516 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:02.170519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:02.171078 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:02.171084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:02.171571 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:02.171573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:02.171632 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:02.171649 (+ 17us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:52:02.605720 (+434071us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 14:52:02.605884 (+ 164us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.34.126:35581: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":121,"threads_started":1}
I20260619 14:52:02.606750 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 25029
2026-06-19T14:52:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (696 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (105 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (107 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:52:02 dist-test-slave-3qdh krb5kdc[25137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:52:02 dist-test-slave-3qdh krb5kdc[25137](info): set up 2 sockets
Jun 19 14:52:02 dist-test-slave-3qdh krb5kdc[25137](info): commencing operation
krb5kdc: starting...
W20260619 14:52:03.816561 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:52:03 dist-test-slave-3qdh krb5kdc[25137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:52:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:52:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:52:03.930532 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:35497
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:33969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:35497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:04.036229 25150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:04.036538 25150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:04.036628 25150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:04.040788 25150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:04.040874 25150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:04.040923 25150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:04.040966 25150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:04.041002 25150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:52:04.046041 25150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:33969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:35497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:35497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:04.047008 25150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:04.047968 25150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:04.055939 25158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:04.055918 25156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:04.055924 25155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:04.056355 25150 server_base.cc:1034] running on GCE node
I20260619 14:52:04.056869 25150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:04.057869 25150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:04.059054 25150 hybrid_clock.cc:648] HybridClock initialized: now 1781880724059038 us; error 33 us; skew 500 ppm
Jun 19 14:52:04 dist-test-slave-3qdh krb5kdc[25137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880724, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:04.066208 25150 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:52:04.067155 25150 webserver.cc:460] Webserver started at http://127.19.34.126:37755/ using document root <none> and password file <none>
I20260619 14:52:04.067696 25150 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:04.067799 25150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:04.068038 25150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:04.070060 25150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "22f787f88cee414daa400baad2971cff"
format_stamp: "Formatted at 2026-06-19 14:52:04 on dist-test-slave-3qdh"
server_key: "e304493306d54e9533695bfa03a4a071"
server_key_iv: "b53d32beb590049ba3d7783b5124d8df"
server_key_version: "encryptionkey@0"
I20260619 14:52:04.070616 25150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "22f787f88cee414daa400baad2971cff"
format_stamp: "Formatted at 2026-06-19 14:52:04 on dist-test-slave-3qdh"
server_key: "e304493306d54e9533695bfa03a4a071"
server_key_iv: "b53d32beb590049ba3d7783b5124d8df"
server_key_version: "encryptionkey@0"
I20260619 14:52:04.074128 25150 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:52:04.076424 25165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:04.077574 25150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:04.077747 25150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "22f787f88cee414daa400baad2971cff"
format_stamp: "Formatted at 2026-06-19 14:52:04 on dist-test-slave-3qdh"
server_key: "e304493306d54e9533695bfa03a4a071"
server_key_iv: "b53d32beb590049ba3d7783b5124d8df"
server_key_version: "encryptionkey@0"
I20260619 14:52:04.077852 25150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:04.099174 25150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:04.101753 25150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:04.101948 25150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:04.108738 25150 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:35497
I20260619 14:52:04.108764 25217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:35497 every 8 connection(s)
I20260619 14:52:04.109597 25150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:52:04.112066 25218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:04.115842 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 25150
I20260619 14:52:04.115968 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:52:04.116241 19593 external_mini_cluster.cc:1428] Setting key c92e63192cff64bf194371d0298e8a5b
I20260619 14:52:04.117576 25218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff: Bootstrap starting.
I20260619 14:52:04.119692 25218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:04.120445 25218 log.cc:824] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:04.122116 25218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff: No bootstrap required, opened a new log
Jun 19 14:52:04 dist-test-slave-3qdh krb5kdc[25137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:52:04.125922 25218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f787f88cee414daa400baad2971cff" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 35497 } }
I20260619 14:52:04.126159 25218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:04.126245 25218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22f787f88cee414daa400baad2971cff, State: Initialized, Role: FOLLOWER
I20260619 14:52:04.126703 25218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f787f88cee414daa400baad2971cff" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 35497 } }
I20260619 14:52:04.126835 25218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:52:04.126928 25218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:52:04.127043 25218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:04.128079 25218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f787f88cee414daa400baad2971cff" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 35497 } }
I20260619 14:52:04.128453 25218 leader_election.cc:302] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22f787f88cee414daa400baad2971cff; no voters:
I20260619 14:52:04.128767 25218 leader_election.cc:288] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:52:04.128888 25223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:04.129195 25223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff [term 1 LEADER]: Becoming Leader. State: Replica: 22f787f88cee414daa400baad2971cff, State: Running, Role: LEADER
I20260619 14:52:04.129519 25223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f787f88cee414daa400baad2971cff" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 35497 } }
I20260619 14:52:04.130016 25218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:04.130719 25223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22f787f88cee414daa400baad2971cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f787f88cee414daa400baad2971cff" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 35497 } } }
I20260619 14:52:04.130790 25224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22f787f88cee414daa400baad2971cff. Latest consensus state: current_term: 1 leader_uuid: "22f787f88cee414daa400baad2971cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22f787f88cee414daa400baad2971cff" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 35497 } } }
I20260619 14:52:04.130899 25223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:04.130998 25224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:04.131726 25221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:04.117620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36154 (local address 127.19.34.126:35497)
0619 14:52:04.118028 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:04.118042 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:04.118075 (+ 33us) server_negotiation.cc:407] Connection header received
0619 14:52:04.118692 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:04.118707 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:04.119006 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:04.119381 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:04.120154 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:04.121369 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:04.122180 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:04.122705 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:04.122870 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:04.125554 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:04.125559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:04.125570 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:04.125594 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:04.127512 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:04.128543 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:04.128550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:04.128556 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:04.128721 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:04.129080 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:04.129083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:04.129085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:04.129565 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:04.130378 (+ 813us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:04.130534 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 14:52:04.130828 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":271,"thread_start_us":117,"threads_started":1}
W20260619 14:52:04.134426 25238 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:52:04.134521 25238 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:52:04.134661 25232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:52:04.135552 25232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:52:04.140682 25232 catalog_manager.cc:1347] Generated new cluster ID: ef5978a894704d5a9e286a6f3f0d4c61
I20260619 14:52:04.140749 25232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:52:04.149855 25232 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:52:04.150934 25232 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:52:04.163573 25232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22f787f88cee414daa400baad2971cff: Generated new TSK 0
I20260619 14:52:04.164304 25232 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.34.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
I20260619 14:52:04.194716 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:35497
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:33969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:04.298661 25244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:04.298874 25244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:04.298913 25244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:04.302781 25244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:04.302829 25244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:04.302881 25244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:52:04.307747 25244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:33969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:35497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:04.308691 25244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:04.309547 25244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:04.318310 25252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:04.318300 25249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:04.318418 25244 server_base.cc:1034] running on GCE node
W20260619 14:52:04.318300 25250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:04.318874 25244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:04.319414 25244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:04.320573 25244 hybrid_clock.cc:648] HybridClock initialized: now 1781880724320535 us; error 50 us; skew 500 ppm
Jun 19 14:52:04 dist-test-slave-3qdh krb5kdc[25137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880724, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:04.323144 25244 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:52:04.323931 25244 webserver.cc:460] Webserver started at http://127.19.34.65:35321/ using document root <none> and password file <none>
I20260619 14:52:04.324501 25244 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:04.324565 25244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:04.324810 25244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:04.326650 25244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "c93dfef08754463cbc63074e4ee1effd"
format_stamp: "Formatted at 2026-06-19 14:52:04 on dist-test-slave-3qdh"
server_key: "859f95bd88672fab3f57204e48fd5ed6"
server_key_iv: "eee9e29c83682687ada060010a3b6343"
server_key_version: "encryptionkey@0"
I20260619 14:52:04.327185 25244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "c93dfef08754463cbc63074e4ee1effd"
format_stamp: "Formatted at 2026-06-19 14:52:04 on dist-test-slave-3qdh"
server_key: "859f95bd88672fab3f57204e48fd5ed6"
server_key_iv: "eee9e29c83682687ada060010a3b6343"
server_key_version: "encryptionkey@0"
I20260619 14:52:04.330752 25244 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:52:04.333007 25259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:04.334091 25244 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:04.334231 25244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "c93dfef08754463cbc63074e4ee1effd"
format_stamp: "Formatted at 2026-06-19 14:52:04 on dist-test-slave-3qdh"
server_key: "859f95bd88672fab3f57204e48fd5ed6"
server_key_iv: "eee9e29c83682687ada060010a3b6343"
server_key_version: "encryptionkey@0"
I20260619 14:52:04.334326 25244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:04.362003 25244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:04.364683 25244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:04.364882 25244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:04.365439 25244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:04.366199 25244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:04.366242 25244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:04.366305 25244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:04.366336 25244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:04.374907 25244 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:35549
I20260619 14:52:04.374933 25372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:35549 every 8 connection(s)
I20260619 14:52:04.375732 25244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:52:04.380592 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 25244
I20260619 14:52:04.380722 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:52:04.380982 19593 external_mini_cluster.cc:1428] Setting key afb5bf97a24d0581157d0a6462d774fc
Jun 19 14:52:04 dist-test-slave-3qdh krb5kdc[25137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880724, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:52:04.389407 25221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:04.377411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:59255 (local address 127.19.34.126:35497)
0619 14:52:04.377539 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:04.377542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:04.378295 (+ 753us) server_negotiation.cc:407] Connection header received
0619 14:52:04.379607 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:04.379610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:04.379679 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:04.379776 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:04.381062 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:04.381679 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:04.382751 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:04.382975 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:04.383103 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:04.385618 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:04.385624 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:04.385631 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:04.385661 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:04.387282 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:04.387926 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:04.387928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:04.387929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:04.388013 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:04.388411 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:04.388413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:04.388414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:04.388567 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:04.388639 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:04.389209 (+ 570us) server_negotiation.cc:299] Negotiation successful
0619 14:52:04.389298 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:52:04.390523 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:04.377743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:35497 (local address 127.19.34.65:59255)
0619 14:52:04.378162 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:04.378199 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:04.379398 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:04.379942 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:04.379949 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:04.380288 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:04.380926 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:04.380940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:04.381792 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:04.381795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:04.382629 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:04.382638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:04.383134 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:04.383147 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:04.383344 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:04.383911 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:04.383926 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:04.385457 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:04.387400 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:04.387408 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:04.387425 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:04.387816 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:04.388104 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:04.388109 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:04.388112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:04.388321 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:04.388676 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:04.388684 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:04.388974 (+ 290us) client_negotiation.cc:769] Sending connection context
0619 14:52:04.389163 (+ 189us) client_negotiation.cc:240] Negotiation successful
0619 14:52:04.389487 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":119,"threads_started":1}
WARNING: no policy specified for kudu/127.19.34.66@KRBTEST.COM; defaulting to no policy
I20260619 14:52:04.391736 25373 heartbeater.cc:347] Connected to a master server at 127.19.34.126:35497
I20260619 14:52:04.391919 25373 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:04.392441 25373 heartbeater.cc:511] Master 127.19.34.126:35497 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.66@KRBTEST.COM" created.
I20260619 14:52:04.393844 25182 ts_manager.cc:194] Registered new tserver with Master: c93dfef08754463cbc63074e4ee1effd (127.19.34.65:35549)
I20260619 14:52:04.395193 25182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:59255
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
I20260619 14:52:04.407503 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:35497
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:33969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:04.512241 25378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:04.512506 25378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:04.512591 25378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:04.516729 25378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:04.516798 25378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:04.516875 25378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:52:04.521822 25378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:33969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:35497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:04.522729 25378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:04.523641 25378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:04.532927 25384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:04.533057 25378 server_base.cc:1034] running on GCE node
W20260619 14:52:04.532990 25383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:04.532960 25386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:04.533582 25378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:04.534148 25378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:04.535327 25378 hybrid_clock.cc:648] HybridClock initialized: now 1781880724535312 us; error 30 us; skew 500 ppm
Jun 19 14:52:04 dist-test-slave-3qdh krb5kdc[25137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880724, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:04.538089 25378 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:52:04.538962 25378 webserver.cc:460] Webserver started at http://127.19.34.66:41965/ using document root <none> and password file <none>
I20260619 14:52:04.539474 25378 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:04.539531 25378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:04.539743 25378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:04.541671 25378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "807df3389fbf46a9b988b8fc2ec882eb"
format_stamp: "Formatted at 2026-06-19 14:52:04 on dist-test-slave-3qdh"
server_key: "92315892dd7c8e8bdff46120cdc67544"
server_key_iv: "bc101cc54d265e90102389d248a3c23e"
server_key_version: "encryptionkey@0"
I20260619 14:52:04.542217 25378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "807df3389fbf46a9b988b8fc2ec882eb"
format_stamp: "Formatted at 2026-06-19 14:52:04 on dist-test-slave-3qdh"
server_key: "92315892dd7c8e8bdff46120cdc67544"
server_key_iv: "bc101cc54d265e90102389d248a3c23e"
server_key_version: "encryptionkey@0"
I20260619 14:52:04.545588 25378 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:52:04.547727 25393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:04.548902 25378 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:04.549022 25378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "807df3389fbf46a9b988b8fc2ec882eb"
format_stamp: "Formatted at 2026-06-19 14:52:04 on dist-test-slave-3qdh"
server_key: "92315892dd7c8e8bdff46120cdc67544"
server_key_iv: "bc101cc54d265e90102389d248a3c23e"
server_key_version: "encryptionkey@0"
I20260619 14:52:04.549114 25378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:04.586331 25378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:04.589413 25378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:04.589641 25378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:04.590221 25378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:04.590979 25378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:04.591023 25378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:04.591092 25378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:04.591136 25378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:04.599308 25378 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:46571
I20260619 14:52:04.599334 25506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:46571 every 8 connection(s)
I20260619 14:52:04.600143 25378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:52:04.602965 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 25378
I20260619 14:52:04.603096 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:52:04.603406 19593 external_mini_cluster.cc:1428] Setting key b81b72b8f756a4a1f5de4b0ae7ec5f6e
Jun 19 14:52:04 dist-test-slave-3qdh krb5kdc[25137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880724, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.34.67@KRBTEST.COM; defaulting to no policy
I20260619 14:52:04.613976 25221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:04.601776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:43441 (local address 127.19.34.126:35497)
0619 14:52:04.601907 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:04.601912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:04.602641 (+ 729us) server_negotiation.cc:407] Connection header received
0619 14:52:04.604266 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:04.604269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:04.604340 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:04.604418 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:04.605863 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:04.606401 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:04.607461 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:04.607707 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:04.607787 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:04.610507 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:04.610511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:04.610513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:04.610534 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:04.611849 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:04.612416 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:04.612419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:04.612421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:04.612508 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:04.612838 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:04.612842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:04.612844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:04.613013 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:04.613114 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:04.613703 (+ 589us) server_negotiation.cc:299] Negotiation successful
0619 14:52:04.613815 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.67@KRBTEST.COM" created.
I20260619 14:52:04.614835 25509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:04.602136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:35497 (local address 127.19.34.66:43441)
0619 14:52:04.602508 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:04.602543 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:04.604065 (+ 1522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:04.604569 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:04.604575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:04.605057 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:04.605704 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:04.605725 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:04.606537 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:04.606540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:04.607355 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:04.607364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:04.607864 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:04.607870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:04.608092 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:04.608792 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:04.608816 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:04.610349 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:04.611986 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:04.611991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:04.612002 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:04.612293 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:04.612609 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:04.612614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:04.612616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:04.612746 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:04.613113 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:04.613119 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:04.613424 (+ 305us) client_negotiation.cc:769] Sending connection context
0619 14:52:04.613650 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 14:52:04.613934 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":102,"threads_started":1}
I20260619 14:52:04.615880 25507 heartbeater.cc:347] Connected to a master server at 127.19.34.126:35497
I20260619 14:52:04.616098 25507 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:04.616591 25507 heartbeater.cc:511] Master 127.19.34.126:35497 requested a full tablet report, sending...
I20260619 14:52:04.617547 25182 ts_manager.cc:194] Registered new tserver with Master: 807df3389fbf46a9b988b8fc2ec882eb (127.19.34.66:46571)
I20260619 14:52:04.618062 25182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:43441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
I20260619 14:52:04.629025 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:35497
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:33969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:04.734287 25512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:04.734526 25512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:04.734591 25512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:04.738646 25512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:04.738708 25512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:04.738783 25512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:52:04.744064 25512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:33969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:35497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:04.745005 25512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:04.745926 25512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:04.755043 25517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:04.755126 25512 server_base.cc:1034] running on GCE node
W20260619 14:52:04.755057 25520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:04.755050 25518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:04.755648 25512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:04.756173 25512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:04.757323 25512 hybrid_clock.cc:648] HybridClock initialized: now 1781880724757314 us; error 28 us; skew 500 ppm
Jun 19 14:52:04 dist-test-slave-3qdh krb5kdc[25137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880724, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:04.759788 25512 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:52:04.760633 25512 webserver.cc:460] Webserver started at http://127.19.34.67:33089/ using document root <none> and password file <none>
I20260619 14:52:04.761127 25512 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:04.761221 25512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:04.761420 25512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:04.763264 25512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "deb9232016ed45fcba82c4ac253aa30e"
format_stamp: "Formatted at 2026-06-19 14:52:04 on dist-test-slave-3qdh"
server_key: "19d3fe80c0568d7956177b2b8ce1a36d"
server_key_iv: "96a87361e6434ef3d9eea50cd824c92e"
server_key_version: "encryptionkey@0"
I20260619 14:52:04.763775 25512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "deb9232016ed45fcba82c4ac253aa30e"
format_stamp: "Formatted at 2026-06-19 14:52:04 on dist-test-slave-3qdh"
server_key: "19d3fe80c0568d7956177b2b8ce1a36d"
server_key_iv: "96a87361e6434ef3d9eea50cd824c92e"
server_key_version: "encryptionkey@0"
I20260619 14:52:04.767053 25512 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:52:04.769186 25527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:04.770243 25512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:04.770358 25512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "deb9232016ed45fcba82c4ac253aa30e"
format_stamp: "Formatted at 2026-06-19 14:52:04 on dist-test-slave-3qdh"
server_key: "19d3fe80c0568d7956177b2b8ce1a36d"
server_key_iv: "96a87361e6434ef3d9eea50cd824c92e"
server_key_version: "encryptionkey@0"
I20260619 14:52:04.770449 25512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:04.813666 25512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:04.817584 25512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:04.817778 25512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:04.818377 25512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:04.819370 25512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:04.819415 25512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:04.819453 25512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:04.819474 25512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:04.827286 25512 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:38375
I20260619 14:52:04.827318 25640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:38375 every 8 connection(s)
I20260619 14:52:04.828068 25512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:52:04.834972 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 25512
I20260619 14:52:04.835078 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:52:04.835340 19593 external_mini_cluster.cc:1428] Setting key 33f9d4aaea7ca7537c3d5101a6cb8947
Jun 19 14:52:04 dist-test-slave-3qdh krb5kdc[25137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880724, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:52:04.840476 25221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:04.829600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:37257 (local address 127.19.34.126:35497)
0619 14:52:04.829771 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:04.829778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:04.830547 (+ 769us) server_negotiation.cc:407] Connection header received
0619 14:52:04.831919 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:04.831923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:04.831980 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:04.832082 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:04.833266 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:04.833721 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:04.834741 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:04.834963 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:04.835074 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:04.837675 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:04.837678 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:04.837680 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:04.837702 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:04.838927 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:04.839328 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:04.839330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:04.839331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:04.839393 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:04.839641 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:04.839643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:04.839644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:04.839782 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:04.839847 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:04.840236 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 14:52:04.840362 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 14:52:04.841176 25643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:04.830033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:35497 (local address 127.19.34.67:37257)
0619 14:52:04.830421 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:04.830458 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:04.831718 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:04.832256 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:04.832264 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:04.832728 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:04.833140 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:04.833152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:04.833830 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:04.833833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:04.834587 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:04.834594 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:04.835104 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:04.835108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:04.835274 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:04.835832 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:04.835863 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:04.837522 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:04.839015 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:04.839018 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:04.839028 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:04.839250 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:04.839462 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:04.839464 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:04.839465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:04.839573 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:04.839857 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:04.839860 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:04.840047 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 14:52:04.840193 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 14:52:04.840461 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":124,"threads_started":1}
I20260619 14:52:04.842021 25641 heartbeater.cc:347] Connected to a master server at 127.19.34.126:35497
I20260619 14:52:04.842178 25641 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:04.842505 25641 heartbeater.cc:511] Master 127.19.34.126:35497 requested a full tablet report, sending...
I20260619 14:52:04.843277 25182 ts_manager.cc:194] Registered new tserver with Master: deb9232016ed45fcba82c4ac253aa30e (127.19.34.67:38375)
I20260619 14:52:04.843703 25182 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:37257
I20260619 14:52:04.848769 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/test-admin.keytab.
Jun 19 14:52:04 dist-test-slave-3qdh krb5kdc[25137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:04.863718 19593 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 14:52:04.866074 25648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:04.864850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57524 (local address 127.19.34.65:35549)
0619 14:52:04.865135 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:04.865148 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:04.865168 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:52:04.865221 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:04.865225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:04.865370 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:04.865476 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:04.865879 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57524: BlockingRecv error: recv got EOF from 127.0.0.1:57524 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":122,"threads_started":1}
W20260619 14:52:04.866452 19593 init.cc:244] Error while opening credential cache '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/krb5cc)
Jun 19 14:52:04 dist-test-slave-3qdh krb5kdc[25137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:04.867410 19593 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 14:52:04 dist-test-slave-3qdh krb5kdc[25137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:04.868284 19593 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 14:52:04 dist-test-slave-3qdh krb5kdc[25137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.65@KRBTEST.COM
I20260619 14:52:04.876964 25648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:04.869074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57536 (local address 127.19.34.65:35549)
0619 14:52:04.869183 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:04.869186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:04.869280 (+ 94us) server_negotiation.cc:407] Connection header received
0619 14:52:04.869362 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:04.869363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:04.869395 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:04.869454 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:04.869877 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:04.870588 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:04.871291 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:04.871585 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:04.871654 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:04.873819 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:04.873827 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:04.873837 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:04.873856 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:04.875405 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:04.875779 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:04.875784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:04.875785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:04.875857 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:04.876080 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:04.876082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:04.876083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:04.876374 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:04.876523 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:04.876686 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 14:52:04.876839 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260619 14:52:04.880005 25648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:04.879202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57550 (local address 127.19.34.65:35549)
0619 14:52:04.879322 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:04.879325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:04.879376 (+ 51us) server_negotiation.cc:407] Connection header received
0619 14:52:04.879464 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:04.879465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:04.879513 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:04.879574 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:04.879933 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57550: BlockingRecv error: recv got EOF from 127.0.0.1:57550 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260619 14:52:04.880110 19593 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/krb5cc' of type FILE
Jun 19 14:52:04 dist-test-slave-3qdh krb5kdc[25137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:04.881044 19593 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 14:52:04 dist-test-slave-3qdh krb5kdc[25137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:04.881844 19593 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 14:52:04 dist-test-slave-3qdh krb5kdc[25137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.65@KRBTEST.COM
I20260619 14:52:04.888720 25648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:04.882478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.19.34.65:35549)
0619 14:52:04.882572 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:04.882575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:04.882686 (+ 111us) server_negotiation.cc:407] Connection header received
0619 14:52:04.882765 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:04.882766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:04.882793 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:04.882838 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:04.883195 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:04.883610 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:04.884242 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:04.884500 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:04.884567 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:04.886357 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:04.886359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:04.886360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:04.886377 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:04.887581 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:04.887902 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:04.887904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:04.887905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:04.887966 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:04.888169 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:04.888170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:04.888171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:04.888290 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:04.888430 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:04.888568 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 14:52:04.888641 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 14:52:04.890751 25648 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:04.889995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57572 (local address 127.19.34.65:35549)
0619 14:52:04.890109 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:04.890113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:04.890198 (+ 85us) server_negotiation.cc:407] Connection header received
0619 14:52:04.890277 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:04.890278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:04.890305 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:04.890358 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:04.890686 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57572: BlockingRecv error: recv got EOF from 127.0.0.1:57572 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 14:52:04.890846 19593 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880697016847-19593-0/krb5kdc/krb5cc' of type FILE
Jun 19 14:52:04 dist-test-slave-3qdh krb5kdc[25137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:04.891718 19593 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 14:52:04 dist-test-slave-3qdh krb5kdc[25137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:04.892536 19593 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 14:52:04 dist-test-slave-3qdh krb5kdc[25137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.65@KRBTEST.COM
I20260619 14:52:04.899636 25648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:04.893208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57582 (local address 127.19.34.65:35549)
0619 14:52:04.893314 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:04.893317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:04.893343 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:52:04.893418 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:04.893420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:04.893445 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:04.893499 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:04.893919 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:04.894347 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:04.895058 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:04.895339 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:04.895414 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:04.897373 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:04.897375 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:04.897377 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:04.897395 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:04.898448 (+ 1053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:04.898791 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:04.898793 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:04.898794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:04.898852 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:04.899043 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:04.899045 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:04.899046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:04.899160 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:04.899248 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:04.899411 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 14:52:04.899506 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:52:04.900391 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 25244
I20260619 14:52:04.905479 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 25378
I20260619 14:52:04.910286 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 25512
I20260619 14:52:04.914927 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 25150
2026-06-19T14:52:04Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2097 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:52:04 dist-test-slave-3qdh krb5kdc[25661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:52:04 dist-test-slave-3qdh krb5kdc[25661](info): set up 2 sockets
Jun 19 14:52:04 dist-test-slave-3qdh krb5kdc[25661](info): commencing operation
krb5kdc: starting...
W20260619 14:52:05.876430 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:52:05 dist-test-slave-3qdh krb5kdc[25661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:52:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:52:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.126.keytab.
Entry for principal oryx/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.126.keytab.
I20260619 14:52:05.983340 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:44099
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.126.keytab
--principal=oryx/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:44099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:06.083324 25674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:06.083571 25674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:06.083652 25674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:06.087397 25674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:06.087477 25674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:06.087513 25674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:06.087546 25674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:06.087577 25674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:52:06.092514 25674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:44099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.126.keytab
--principal=oryx/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:44099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:06.093426 25674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:06.094309 25674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:06.101856 25679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:06.101851 25680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:06.101926 25682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:06.102047 25674 server_base.cc:1034] running on GCE node
I20260619 14:52:06.102610 25674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:06.103513 25674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:06.104686 25674 hybrid_clock.cc:648] HybridClock initialized: now 1781880726104666 us; error 39 us; skew 500 ppm
Jun 19 14:52:06 dist-test-slave-3qdh krb5kdc[25661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880726, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:06.107206 25674 init.cc:377] Logged in from keytab as oryx/127.19.34.126@KRBTEST.COM (short username oryx)
I20260619 14:52:06.107918 25674 webserver.cc:460] Webserver started at http://127.19.34.126:37497/ using document root <none> and password file <none>
I20260619 14:52:06.108448 25674 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:06.108527 25674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:06.108722 25674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:06.110386 25674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc"
format_stamp: "Formatted at 2026-06-19 14:52:06 on dist-test-slave-3qdh"
server_key: "37d236652b2d2c69445633ef5ee358c1"
server_key_iv: "102555933d7cc57143bb1e44fb1fc2ec"
server_key_version: "encryptionkey@0"
I20260619 14:52:06.110867 25674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc"
format_stamp: "Formatted at 2026-06-19 14:52:06 on dist-test-slave-3qdh"
server_key: "37d236652b2d2c69445633ef5ee358c1"
server_key_iv: "102555933d7cc57143bb1e44fb1fc2ec"
server_key_version: "encryptionkey@0"
I20260619 14:52:06.113917 25674 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:52:06.115983 25689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:06.117019 25674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:06.117141 25674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc"
format_stamp: "Formatted at 2026-06-19 14:52:06 on dist-test-slave-3qdh"
server_key: "37d236652b2d2c69445633ef5ee358c1"
server_key_iv: "102555933d7cc57143bb1e44fb1fc2ec"
server_key_version: "encryptionkey@0"
I20260619 14:52:06.117240 25674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:06.133953 25674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:06.137071 25674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:06.137215 25674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:06.142925 25674 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:44099
I20260619 14:52:06.143178 25741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:44099 every 8 connection(s)
I20260619 14:52:06.143733 25674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:52:06.146152 25742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:06.148020 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 25674
I20260619 14:52:06.148137 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:52:06.148432 19593 external_mini_cluster.cc:1428] Setting key 1df81c4f010706436e7c19c574c972eb
I20260619 14:52:06.151543 25742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc: Bootstrap starting.
I20260619 14:52:06.153457 25742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:06.154095 25742 log.cc:824] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:06.155881 25742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc: No bootstrap required, opened a new log
Jun 19 14:52:06 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM
I20260619 14:52:06.159089 25742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } }
I20260619 14:52:06.159377 25742 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:06.159452 25742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6af1f893a84c1d81d1a3c0ec3dc9bc, State: Initialized, Role: FOLLOWER
I20260619 14:52:06.159926 25742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } }
I20260619 14:52:06.160094 25742 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:52:06.160180 25742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:52:06.160297 25742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:06.161353 25742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } }
I20260619 14:52:06.161693 25742 leader_election.cc:302] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e6af1f893a84c1d81d1a3c0ec3dc9bc; no voters:
I20260619 14:52:06.162001 25742 leader_election.cc:288] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:52:06.162011 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:06.149676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53584 (local address 127.19.34.126:44099)
0619 14:52:06.150028 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:06.150041 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:06.150070 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:52:06.150779 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:06.150794 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:06.151134 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:06.151457 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:06.152114 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:06.153091 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:06.153915 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:06.154375 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:06.154537 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:06.157018 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:06.157022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:06.157032 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:06.157052 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:06.158876 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:06.159362 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:06.159367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:06.159372 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:06.159505 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:06.159790 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:06.159793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:06.159795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:06.160200 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:06.160505 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:06.160780 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 14:52:06.161106 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":114,"threads_started":1}
I20260619 14:52:06.162103 25747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:06.162292 25747 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [term 1 LEADER]: Becoming Leader. State: Replica: 4e6af1f893a84c1d81d1a3c0ec3dc9bc, State: Running, Role: LEADER
I20260619 14:52:06.162616 25747 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } }
I20260619 14:52:06.163208 25742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:06.163657 25747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } }
I20260619 14:52:06.163833 25747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:06.164238 25754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:52:06.165025 25748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e6af1f893a84c1d81d1a3c0ec3dc9bc. Latest consensus state: current_term: 1 leader_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } }
I20260619 14:52:06.165203 25748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:06.167259 25754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:52:06.172286 25754 catalog_manager.cc:1347] Generated new cluster ID: 09ee1b26ed4542908dcf941d1d1f6802
I20260619 14:52:06.172413 25754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:52:06.189015 25754 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:52:06.189779 25754 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:52:06.195806 25754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc: Generated new TSK 0
I20260619 14:52:06.196420 25754 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.34.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.34.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.34.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.65.keytab.
Entry for principal oryx/127.19.34.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.65.keytab.
I20260619 14:52:06.228984 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44099
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.65.keytab
--principal=oryx/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:06.332916 25768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:06.333134 25768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:06.333173 25768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:06.336818 25768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:06.336863 25768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:06.336951 25768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:52:06.341440 25768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.65.keytab
--principal=oryx/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:06.342258 25768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:06.343106 25768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:06.351868 25773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:06.351941 25776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:06.351941 25774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:06.352051 25768 server_base.cc:1034] running on GCE node
I20260619 14:52:06.352454 25768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:06.352941 25768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:06.354113 25768 hybrid_clock.cc:648] HybridClock initialized: now 1781880726354089 us; error 39 us; skew 500 ppm
Jun 19 14:52:06 dist-test-slave-3qdh krb5kdc[25661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880726, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:06.356802 25768 init.cc:377] Logged in from keytab as oryx/127.19.34.65@KRBTEST.COM (short username oryx)
I20260619 14:52:06.357486 25768 webserver.cc:460] Webserver started at http://127.19.34.65:39301/ using document root <none> and password file <none>
I20260619 14:52:06.357934 25768 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:06.357985 25768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:06.358120 25768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:06.359910 25768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "dc422cbf78144431af76287ded9cdf97"
format_stamp: "Formatted at 2026-06-19 14:52:06 on dist-test-slave-3qdh"
server_key: "e6a8db285ac2817ead64c08ccaa705fd"
server_key_iv: "8522fdc240c05212dde6fe281fe7a3b2"
server_key_version: "encryptionkey@0"
I20260619 14:52:06.360430 25768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "dc422cbf78144431af76287ded9cdf97"
format_stamp: "Formatted at 2026-06-19 14:52:06 on dist-test-slave-3qdh"
server_key: "e6a8db285ac2817ead64c08ccaa705fd"
server_key_iv: "8522fdc240c05212dde6fe281fe7a3b2"
server_key_version: "encryptionkey@0"
I20260619 14:52:06.363631 25768 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 14:52:06.365854 25783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:06.367007 25768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:52:06.367086 25768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "dc422cbf78144431af76287ded9cdf97"
format_stamp: "Formatted at 2026-06-19 14:52:06 on dist-test-slave-3qdh"
server_key: "e6a8db285ac2817ead64c08ccaa705fd"
server_key_iv: "8522fdc240c05212dde6fe281fe7a3b2"
server_key_version: "encryptionkey@0"
I20260619 14:52:06.367154 25768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:06.376734 25768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:06.379336 25768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:06.379457 25768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:06.379894 25768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:06.380602 25768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:06.380645 25768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:06.380678 25768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:06.380692 25768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:06.388521 25768 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:44803
I20260619 14:52:06.388556 25896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:44803 every 8 connection(s)
I20260619 14:52:06.389206 25768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:52:06.394261 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 25768
I20260619 14:52:06.394366 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:52:06.394629 19593 external_mini_cluster.cc:1428] Setting key cc82f10270e8ab54874eeaa6e08d2fd7
Jun 19 14:52:06 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880726, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM
I20260619 14:52:06.401284 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:06.390632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:40869 (local address 127.19.34.126:44099)
0619 14:52:06.390749 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:06.390752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:06.391420 (+ 668us) server_negotiation.cc:407] Connection header received
0619 14:52:06.392542 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:06.392544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:06.392586 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:06.392653 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:06.393560 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:06.394045 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:06.395435 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:06.395768 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:06.395901 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:06.398255 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:06.398260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:06.398263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:06.398287 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:06.399545 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:06.399977 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:06.399981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:06.399982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:06.400052 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:06.400377 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:06.400380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:06.400382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:06.400532 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:06.400626 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:06.401024 (+ 398us) server_negotiation.cc:299] Negotiation successful
0619 14:52:06.401121 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:52:06.402077 25899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:06.390916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.65:40869)
0619 14:52:06.391288 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:06.391320 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:06.392371 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:06.392784 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:06.392789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:06.393067 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:06.393450 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:06.393461 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:06.394169 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:06.394172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:06.395328 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:06.395337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:06.395936 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:06.395939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:06.396140 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:06.396653 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:06.396672 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:06.398109 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:06.399652 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:06.399657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:06.399668 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:06.399885 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:06.400134 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:06.400137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:06.400139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:06.400264 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:06.400626 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:06.400632 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:06.400825 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 14:52:06.400973 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 14:52:06.401237 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for oryx/127.19.34.66@KRBTEST.COM; defaulting to no policy
I20260619 14:52:06.403275 25897 heartbeater.cc:347] Connected to a master server at 127.19.34.126:44099
I20260619 14:52:06.403478 25897 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:06.403924 25897 heartbeater.cc:511] Master 127.19.34.126:44099 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.34.66@KRBTEST.COM" created.
I20260619 14:52:06.405344 25706 ts_manager.cc:194] Registered new tserver with Master: dc422cbf78144431af76287ded9cdf97 (127.19.34.65:44803)
I20260619 14:52:06.406404 25706 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:40869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.34.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.66.keytab.
Entry for principal oryx/127.19.34.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.66.keytab.
I20260619 14:52:06.418607 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44099
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.66.keytab
--principal=oryx/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:06.517978 25902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:06.518188 25902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:06.518224 25902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:06.521667 25902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:06.521713 25902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:06.521761 25902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:52:06.526402 25902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.66.keytab
--principal=oryx/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:06.527184 25902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:06.527963 25902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:06.536127 25908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:06.536216 25902 server_base.cc:1034] running on GCE node
W20260619 14:52:06.536197 25910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:06.536437 25907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:06.536787 25902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:06.537391 25902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:06.538552 25902 hybrid_clock.cc:648] HybridClock initialized: now 1781880726538538 us; error 37 us; skew 500 ppm
Jun 19 14:52:06 dist-test-slave-3qdh krb5kdc[25661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880726, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:06.541102 25902 init.cc:377] Logged in from keytab as oryx/127.19.34.66@KRBTEST.COM (short username oryx)
I20260619 14:52:06.541857 25902 webserver.cc:460] Webserver started at http://127.19.34.66:35087/ using document root <none> and password file <none>
I20260619 14:52:06.542320 25902 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:06.542399 25902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:06.542593 25902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:06.544245 25902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "bd8e60ecb8744ed2ab3e8a58713459dd"
format_stamp: "Formatted at 2026-06-19 14:52:06 on dist-test-slave-3qdh"
server_key: "8066407286d9d4427bb13a74ecaf4a12"
server_key_iv: "c4cc1b3635e8dcacecd6e0142e0676c1"
server_key_version: "encryptionkey@0"
I20260619 14:52:06.544855 25902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "bd8e60ecb8744ed2ab3e8a58713459dd"
format_stamp: "Formatted at 2026-06-19 14:52:06 on dist-test-slave-3qdh"
server_key: "8066407286d9d4427bb13a74ecaf4a12"
server_key_iv: "c4cc1b3635e8dcacecd6e0142e0676c1"
server_key_version: "encryptionkey@0"
I20260619 14:52:06.547962 25902 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:52:06.550099 25917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:06.551092 25902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:06.551231 25902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "bd8e60ecb8744ed2ab3e8a58713459dd"
format_stamp: "Formatted at 2026-06-19 14:52:06 on dist-test-slave-3qdh"
server_key: "8066407286d9d4427bb13a74ecaf4a12"
server_key_iv: "c4cc1b3635e8dcacecd6e0142e0676c1"
server_key_version: "encryptionkey@0"
I20260619 14:52:06.551347 25902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:06.559899 25902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:06.562094 25902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:06.562251 25902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:06.562705 25902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:06.563423 25902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:06.563493 25902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:06.563544 25902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:06.563606 25902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:06.571630 25902 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:39517
I20260619 14:52:06.571660 26030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:39517 every 8 connection(s)
I20260619 14:52:06.572403 25902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:52:06.573248 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 25902
I20260619 14:52:06.573350 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:52:06.573628 19593 external_mini_cluster.cc:1428] Setting key aa4c6a58acf3fe68519b105ec6856038
Jun 19 14:52:06 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880726, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.19.34.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.34.67@KRBTEST.COM" created.
I20260619 14:52:06.586099 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:06.574088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:38269 (local address 127.19.34.126:44099)
0619 14:52:06.574227 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:06.574234 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:06.575026 (+ 792us) server_negotiation.cc:407] Connection header received
0619 14:52:06.576826 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:06.576831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:06.576888 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:06.576976 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:06.577904 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:06.578372 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:06.579911 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:06.580088 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:06.580153 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:06.582745 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:06.582749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:06.582752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:06.582779 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:06.584178 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:06.584639 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:06.584644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:06.584646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:06.584720 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:06.585061 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:06.585064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:06.585066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:06.585237 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:06.585341 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:06.585875 (+ 534us) server_negotiation.cc:299] Negotiation successful
0619 14:52:06.585978 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:52:06.587101 26033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:06.574449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.66:38269)
0619 14:52:06.574898 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:06.574935 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:06.576622 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:06.577133 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:06.577138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:06.577430 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:06.577793 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:06.577803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:06.578491 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:06.578496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:06.579800 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:06.579813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:06.580240 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:06.580247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:06.580494 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:06.581007 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:06.581026 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:06.582598 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:06.584291 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:06.584313 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:06.584324 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:06.584545 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:06.584823 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:06.584827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:06.584829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:06.584960 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:06.585339 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:06.585346 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:06.585615 (+ 269us) client_negotiation.cc:769] Sending connection context
0619 14:52:06.585802 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 14:52:06.586144 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":96,"threads_started":1}
I20260619 14:52:06.588135 26031 heartbeater.cc:347] Connected to a master server at 127.19.34.126:44099
I20260619 14:52:06.588371 26031 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:06.588871 26031 heartbeater.cc:511] Master 127.19.34.126:44099 requested a full tablet report, sending...
I20260619 14:52:06.589962 25706 ts_manager.cc:194] Registered new tserver with Master: bd8e60ecb8744ed2ab3e8a58713459dd (127.19.34.66:39517)
I20260619 14:52:06.590507 25706 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:38269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.34.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.67.keytab.
Entry for principal oryx/127.19.34.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.67.keytab.
I20260619 14:52:06.601033 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44099
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.67.keytab
--principal=oryx/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:06.703939 26036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:06.704149 26036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:06.704188 26036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:06.707858 26036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:06.707933 26036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:06.707989 26036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:52:06.712853 26036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.67.keytab
--principal=oryx/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:06.713627 26036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:06.714447 26036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:06.723027 26041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:06.723083 26044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:06.723245 26036 server_base.cc:1034] running on GCE node
W20260619 14:52:06.723079 26042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:06.723739 26036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:06.724339 26036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:06.725517 26036 hybrid_clock.cc:648] HybridClock initialized: now 1781880726725498 us; error 27 us; skew 500 ppm
Jun 19 14:52:06 dist-test-slave-3qdh krb5kdc[25661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880726, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:06.728116 26036 init.cc:377] Logged in from keytab as oryx/127.19.34.67@KRBTEST.COM (short username oryx)
I20260619 14:52:06.728963 26036 webserver.cc:460] Webserver started at http://127.19.34.67:44397/ using document root <none> and password file <none>
I20260619 14:52:06.729455 26036 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:06.729540 26036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:06.729733 26036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:06.731549 26036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "8b22c40ce65d4e669cff4d97820eca21"
format_stamp: "Formatted at 2026-06-19 14:52:06 on dist-test-slave-3qdh"
server_key: "b90f3072e35578d2a4de0a7bf017f1e4"
server_key_iv: "e25b62983406fd733436eeba869e7bdf"
server_key_version: "encryptionkey@0"
I20260619 14:52:06.732118 26036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "8b22c40ce65d4e669cff4d97820eca21"
format_stamp: "Formatted at 2026-06-19 14:52:06 on dist-test-slave-3qdh"
server_key: "b90f3072e35578d2a4de0a7bf017f1e4"
server_key_iv: "e25b62983406fd733436eeba869e7bdf"
server_key_version: "encryptionkey@0"
I20260619 14:52:06.735741 26036 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:52:06.738132 26051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:06.739198 26036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:06.739329 26036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "8b22c40ce65d4e669cff4d97820eca21"
format_stamp: "Formatted at 2026-06-19 14:52:06 on dist-test-slave-3qdh"
server_key: "b90f3072e35578d2a4de0a7bf017f1e4"
server_key_iv: "e25b62983406fd733436eeba869e7bdf"
server_key_version: "encryptionkey@0"
I20260619 14:52:06.739426 26036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:06.777920 26036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:06.780198 26036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:06.780380 26036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:06.780851 26036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:06.781523 26036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:06.781585 26036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:06.781641 26036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:06.781670 26036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:06.789369 26036 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:33973
I20260619 14:52:06.789390 26164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:33973 every 8 connection(s)
I20260619 14:52:06.790089 26036 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:52:06.795979 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 26036
I20260619 14:52:06.796113 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:52:06.796422 19593 external_mini_cluster.cc:1428] Setting key 93251a58c97f52f88ef42051da3ddbce
Jun 19 14:52:06 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880726, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM
I20260619 14:52:06.802485 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:06.791717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:34313 (local address 127.19.34.126:44099)
0619 14:52:06.791830 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:06.791833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:06.792591 (+ 758us) server_negotiation.cc:407] Connection header received
0619 14:52:06.793725 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:06.793728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:06.793780 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:06.793891 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:06.794972 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:06.795491 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:06.796793 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:06.797112 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:06.797226 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:06.799640 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:06.799643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:06.799645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:06.799665 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:06.801038 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:06.801423 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:06.801426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:06.801427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:06.801488 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:06.801732 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:06.801734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:06.801735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:06.801873 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:06.801939 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:06.802303 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 14:52:06.802385 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:52:06.803130 26167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:06.792053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.67:34313)
0619 14:52:06.792474 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:06.792509 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:06.793542 (+ 1033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:06.794071 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:06.794076 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:06.794415 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:06.794819 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:06.794831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:06.795609 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:06.795615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:06.796676 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:06.796687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:06.797277 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:06.797283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:06.797440 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:06.797989 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:06.798009 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:06.799501 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:06.801122 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:06.801126 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:06.801134 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:06.801347 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:06.801556 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:06.801558 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:06.801559 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:06.801663 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:06.801947 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:06.801951 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:06.802121 (+ 170us) client_negotiation.cc:769] Sending connection context
0619 14:52:06.802260 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 14:52:06.802453 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":92,"threads_started":1}
I20260619 14:52:06.804085 26165 heartbeater.cc:347] Connected to a master server at 127.19.34.126:44099
I20260619 14:52:06.804282 26165 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:06.804687 26165 heartbeater.cc:511] Master 127.19.34.126:44099 requested a full tablet report, sending...
I20260619 14:52:06.805603 25706 ts_manager.cc:194] Registered new tserver with Master: 8b22c40ce65d4e669cff4d97820eca21 (127.19.34.67:33973)
I20260619 14:52:06.806107 25706 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:34313
I20260619 14:52:06.809770 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.34.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.34.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.34.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.125.keytab.
Entry for principal oryx/127.19.34.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.125.keytab.
I20260619 14:52:06.834781 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.125:44365
--webserver_interface=127.19.34.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.125.keytab
--principal=oryx/127.19.34.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:44099,127.19.34.125:44365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:06.929028 26170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:06.929219 26170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:06.929284 26170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:06.932857 26170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:06.932911 26170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:06.932929 26170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:06.932976 26170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:06.933003 26170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:52:06.937529 26170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:44099,127.19.34.125:44365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.125.keytab
--principal=oryx/127.19.34.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.125:44365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.34.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
W20260619 14:52:06.937697 26170 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.19.34.126:44099,127.19.34.125:44365'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 14:52:06.938349 26170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:06.939213 26170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:06.946702 26176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:06.946707 26175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:06.946796 26170 server_base.cc:1034] running on GCE node
W20260619 14:52:06.946856 26178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:06.947394 26170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:06.948371 26170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:06.949543 26170 hybrid_clock.cc:648] HybridClock initialized: now 1781880726949525 us; error 44 us; skew 500 ppm
Jun 19 14:52:06 dist-test-slave-3qdh krb5kdc[25661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880726, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:06.952047 26170 init.cc:377] Logged in from keytab as oryx/127.19.34.125@KRBTEST.COM (short username oryx)
I20260619 14:52:06.952883 26170 webserver.cc:460] Webserver started at http://127.19.34.125:38713/ using document root <none> and password file <none>
I20260619 14:52:06.953368 26170 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:06.953464 26170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:06.953658 26170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:06.955514 26170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/data/instance:
uuid: "5454034a11a7476699e9b65bb54329a3"
format_stamp: "Formatted at 2026-06-19 14:52:06 on dist-test-slave-3qdh"
server_key: "bf3cde73928da611d6eafb921b28bd0d"
server_key_iv: "85b613b9e955af758a2ba33e2eb3fcad"
server_key_version: "encryptionkey@0"
I20260619 14:52:06.956053 26170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/wal/instance:
uuid: "5454034a11a7476699e9b65bb54329a3"
format_stamp: "Formatted at 2026-06-19 14:52:06 on dist-test-slave-3qdh"
server_key: "bf3cde73928da611d6eafb921b28bd0d"
server_key_iv: "85b613b9e955af758a2ba33e2eb3fcad"
server_key_version: "encryptionkey@0"
I20260619 14:52:06.959323 26170 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:52:06.961280 26185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:06.962288 26170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:52:06.962406 26170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/wal
uuid: "5454034a11a7476699e9b65bb54329a3"
format_stamp: "Formatted at 2026-06-19 14:52:06 on dist-test-slave-3qdh"
server_key: "bf3cde73928da611d6eafb921b28bd0d"
server_key_iv: "85b613b9e955af758a2ba33e2eb3fcad"
server_key_version: "encryptionkey@0"
I20260619 14:52:06.962494 26170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:06.974356 26170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:06.977375 26170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:06.977550 26170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:06.983592 26170 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.125:44365
I20260619 14:52:06.983824 26237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.125:44365 every 8 connection(s)
I20260619 14:52:06.984423 26170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/data/info.pb
I20260619 14:52:06.986781 26238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:06.989502 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 26170
I20260619 14:52:06.990125 26238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:52:06 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880726, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.125@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM
Jun 19 14:52:06 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.34.125@KRBTEST.COM
I20260619 14:52:07.002286 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:06.991895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51196 (local address 127.19.34.126:44099)
0619 14:52:06.992033 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:06.992038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:06.992333 (+ 295us) server_negotiation.cc:407] Connection header received
0619 14:52:06.992610 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:06.992613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:06.992658 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:06.992739 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:06.993592 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:06.994253 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:06.995570 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:06.995908 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:06.996023 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:06.998609 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:06.998612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:06.998614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:06.998633 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.000219 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.000766 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.000772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.000773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.000861 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.001233 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.001235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.001237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.001466 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.001579 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.002005 (+ 426us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.002132 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:52:07.002951 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:06.991802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.0.0.1:51196)
0619 14:52:06.992189 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:06.992231 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:06.992413 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:06.992804 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:06.992809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:06.993194 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:06.993479 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:06.993490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:06.994361 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:06.994363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:06.995459 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:06.995471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:06.996054 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:06.996056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:06.996260 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:06.996881 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:06.996902 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:06.998466 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.000353 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.000360 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.000365 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.000677 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.000943 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.000946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.000948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.001125 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.001585 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.001589 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.001757 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.001923 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.002200 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":40,"thread_start_us":146,"threads_started":1}
I20260619 14:52:07.002956 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:06.990880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39346 (local address 127.19.34.125:44365)
0619 14:52:06.991259 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:06.991275 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:06.991309 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:52:06.992046 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:06.992080 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:06.992422 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:06.992656 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:06.993412 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:06.994469 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:06.995570 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:06.996131 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:06.996266 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:06.999048 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:06.999056 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:06.999068 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:06.999089 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.000729 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.001203 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.001210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.001214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.001295 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.001537 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.001542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.001545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.001765 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.001927 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.002144 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.002253 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":252,"spinlock_wait_cycles":1408,"thread_start_us":89,"threads_started":1}
I20260619 14:52:07.004035 26238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880726, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.125@KRBTEST.COM for oryx/127.19.34.125@KRBTEST.COM
I20260619 14:52:07.011247 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.004493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:44365 (local address 127.0.0.1:39360)
0619 14:52:07.004614 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.004629 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.004695 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.004937 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.004940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.005080 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.005150 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.005155 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.005841 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.005844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.006760 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.006767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.007184 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.007187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.007296 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.007608 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.007616 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.008652 (+ 1036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.009957 (+ 1305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.009962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.009965 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.010123 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.010364 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.010368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.010370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.010437 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.010748 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.010750 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.010824 (+ 74us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.010937 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.011052 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":16}
I20260619 14:52:07.011257 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.004579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39360 (local address 127.19.34.125:44365)
0619 14:52:07.004682 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.004685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.004697 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:52:07.004772 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.004775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.004810 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.004928 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.005265 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.005735 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.006861 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.007076 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.007181 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.008766 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.008769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.008771 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.008814 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.009863 (+ 1049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.010220 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.010223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.010224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.010288 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.010514 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.010519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.010521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.010656 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.010748 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.010976 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.011104 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":34}
I20260619 14:52:07.013542 26238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: Bootstrap starting.
I20260619 14:52:07.015378 26238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:07.016002 26238 log.cc:824] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:07.017562 26238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: No bootstrap required, opened a new log
I20260619 14:52:07.020031 26238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } }
I20260619 14:52:07.020176 26238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:07.020205 26238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5454034a11a7476699e9b65bb54329a3, State: Initialized, Role: FOLLOWER
I20260619 14:52:07.020550 26238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } }
I20260619 14:52:07.020885 26244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } } }
I20260619 14:52:07.021054 26244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:52:07.021492 26238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:52:07.025309 26255 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:52:07.025396 26255 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:52:07.026948 26170 master_runner.cc:428] Detected that this master 5454034a11a7476699e9b65bb54329a3 is joining an existing cluster
I20260619 14:52:07.027016 26170 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.34.125:44365
I20260619 14:52:07.035233 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.028739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51206 (local address 127.19.34.126:44099)
0619 14:52:07.029061 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.029064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.029076 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:52:07.029148 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.029150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.029182 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.029261 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.029628 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.030064 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.030827 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.031044 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.031153 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.032077 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.032080 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.032082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.032103 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.033744 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.034087 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.034090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.034092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.034169 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.034422 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.034425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.034426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.034605 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.034691 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.034968 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.035077 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251}
I20260619 14:52:07.035251 26262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.028686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.0.0.1:51206)
0619 14:52:07.028987 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.029000 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.029073 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.029273 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.029276 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.029446 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.029525 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.029530 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.030162 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.030167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.030720 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.030727 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.031158 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.031161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.031261 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.031686 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.031696 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.031950 (+ 254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.033830 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.033834 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.033836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.034001 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.034247 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.034251 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.034253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.034326 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.034696 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.034699 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.034770 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.034900 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.035066 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":35,"thread_start_us":79,"threads_started":1}
I20260619 14:52:07.039458 26261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.028732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:44365 (local address 127.0.0.1:39374)
0619 14:52:07.032495 (+ 3763us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.032508 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.032611 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.032897 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.032902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.033118 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.033442 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.033451 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.034304 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.034306 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.035020 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.035033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.035624 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.035628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.035740 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.036201 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.036211 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.036538 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.038134 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.038139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.038141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.038308 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.038551 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.038554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.038555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.038624 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.038932 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.038935 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.039028 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.039159 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.039287 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3687,"thread_start_us":114,"threads_started":1}
I20260619 14:52:07.039479 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.029039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39374 (local address 127.19.34.125:44365)
0619 14:52:07.029123 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.029126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.032577 (+ 3451us) server_negotiation.cc:407] Connection header received
0619 14:52:07.032721 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.032725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.032760 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.032830 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.033557 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.034214 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.035141 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.035468 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.035532 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.036649 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.036651 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.036653 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.036671 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.038038 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.038393 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.038397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.038399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.038459 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.038700 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.038703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.038704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.038852 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.038935 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.039220 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.039346 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":36}
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880726, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.126@KRBTEST.COM for oryx/127.19.34.125@KRBTEST.COM
I20260619 14:52:07.047813 26263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.039175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:44365 (local address 127.0.0.1:39378)
0619 14:52:07.040119 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.040161 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.040296 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.040609 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.040614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.041018 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.041142 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.041147 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.042010 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.042012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.042641 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.042649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.043119 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.043122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.043231 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.043632 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.043641 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.045033 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.046626 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.046634 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.046637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.046812 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.047035 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.047036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.047037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.047097 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.047433 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.047437 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.047513 (+ 76us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.047603 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.047714 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"thread_start_us":62,"threads_started":1}
I20260619 14:52:07.047902 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.039707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39378 (local address 127.19.34.125:44365)
0619 14:52:07.039946 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.039949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.040228 (+ 279us) server_negotiation.cc:407] Connection header received
0619 14:52:07.040409 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.040411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.040447 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.040505 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.041247 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.041919 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.042739 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.043028 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.043119 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.045156 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.045160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.045163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.045193 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.046540 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.046890 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.046892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.046894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.046961 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.047185 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.047187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.047189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.047353 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.047430 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.047665 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.047748 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
I20260619 14:52:07.049182 25706 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.34.125:44365: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5454034a11a7476699e9b65bb54329a3"
member_type: NON_VOTER
last_known_addr {
host: "127.19.34.125"
port: 44365
}
attrs {
promote: true
}
}
dest_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc"
cas_config_opid_index: -1
I20260619 14:52:07.049752 25706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: NON_VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: true } }
I20260619 14:52:07.051213 26266 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5454034a11a7476699e9b65bb54329a3 (127.19.34.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: NON_VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: true } } }
I20260619 14:52:07.051764 26212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:07.052708 26266 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.34.125:44365
I20260619 14:52:07.052870 26212 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 1 FOLLOWER]: Refusing update from remote peer 4e6af1f893a84c1d81d1a3c0ec3dc9bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 14:52:07.052904 26266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: NON_VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: true } } }
I20260619 14:52:07.052985 26266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:07.053081 26170 master.cc:537] Master@127.19.34.125:44365 shutting down...
I20260619 14:52:07.053449 26266 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: NON_VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:52:07.055387 25690 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 14:52:07.056313 25690 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc -> Peer 5454034a11a7476699e9b65bb54329a3 (127.19.34.125:44365): Couldn't send request to peer 5454034a11a7476699e9b65bb54329a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 14:52:07.057453 26170 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:07.057696 26170 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:07.057813 26170 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: stopping tablet replica
W20260619 14:52:07.060261 26188 connection.cc:441] server connection from 127.0.0.1:39346 torn down before Call kudu.master.MasterService.ListTables from 127.0.0.1:39346 (request call id 12) could send its response
I20260619 14:52:07.061784 26170 master.cc:552] Master@127.19.34.125:44365 shutdown complete.
I20260619 14:52:07.061842 26170 master_runner.cc:305] Clearing existing system tablet
I20260619 14:52:07.062041 26170 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:07.062098 26170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:07.062989 26170 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260619 14:52:07.063701 26270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:07.063942 26170 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:07.064074 26170 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:52:07.064114 26170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/wal
uuid: "5454034a11a7476699e9b65bb54329a3"
format_stamp: "Formatted at 2026-06-19 14:52:06 on dist-test-slave-3qdh"
server_key: "bf3cde73928da611d6eafb921b28bd0d"
server_key_iv: "85b613b9e955af758a2ba33e2eb3fcad"
server_key_version: "encryptionkey@0"
I20260619 14:52:07.065725 26170 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:07.067255 26170 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 14:52:07.067375 26170 log.cc:1196] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 14:52:07.067628 26170 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: Deleting consensus metadata
I20260619 14:52:07.067775 26170 master_runner.cc:315] Copying system tablet from 127.19.34.126:44099
I20260619 14:52:07.068775 26170 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: tablet copy: Beginning tablet copy session from remote peer at address 127.19.34.126:44099
I20260619 14:52:07.075722 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.069246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51210 (local address 127.19.34.126:44099)
0619 14:52:07.069351 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.069355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.069455 (+ 100us) server_negotiation.cc:407] Connection header received
0619 14:52:07.069556 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.069558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.069631 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.069726 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.070057 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.070484 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.071245 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.071426 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.071515 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.072874 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.072878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.072880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.072900 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.074190 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.074584 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.074588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.074590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.074668 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.074951 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.074956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.074958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.075121 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.075223 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.075483 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.075579 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:52:07.075728 26276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.069192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.0.0.1:51210)
0619 14:52:07.069395 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.069407 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.069490 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.069740 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.069743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.069899 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.069961 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.069966 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.070574 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.070578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.071139 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.071146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.071517 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.071520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.071604 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.072407 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.072424 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.072758 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.074305 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.074309 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.074311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.074492 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.074755 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.074759 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.074761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.074860 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.075225 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.075228 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.075320 (+ 92us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.075419 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.075543 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
I20260619 14:52:07.076272 25726 tablet_copy_service.cc:140] P 4e6af1f893a84c1d81d1a3c0ec3dc9bc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5454034a11a7476699e9b65bb54329a3 ({username='oryx', principal='oryx/127.19.34.125@KRBTEST.COM'} at 127.0.0.1:51210)
I20260619 14:52:07.076474 25726 tablet_copy_service.cc:161] P 4e6af1f893a84c1d81d1a3c0ec3dc9bc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5454034a11a7476699e9b65bb54329a3 at {username='oryx', principal='oryx/127.19.34.125@KRBTEST.COM'} at 127.0.0.1:51210: session id = 5454034a11a7476699e9b65bb54329a3-00000000000000000000000000000000
I20260619 14:52:07.077600 25726 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 14:52:07.078740 26170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 14:52:07.080122 26170 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: tablet copy: Starting download of 0 data blocks...
I20260619 14:52:07.080287 26170 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: tablet copy: Starting download of 1 WAL segments...
I20260619 14:52:07.081660 26170 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 14:52:07.082800 25726 tablet_copy_service.cc:342] P 4e6af1f893a84c1d81d1a3c0ec3dc9bc: Request end of tablet copy session 5454034a11a7476699e9b65bb54329a3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.34.125@KRBTEST.COM'} at 127.0.0.1:51210
I20260619 14:52:07.082870 25726 tablet_copy_service.cc:433] P 4e6af1f893a84c1d81d1a3c0ec3dc9bc: ending tablet copy session 5454034a11a7476699e9b65bb54329a3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5454034a11a7476699e9b65bb54329a3
I20260619 14:52:07.088186 26170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:07.089638 26277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:07.089690 26278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:07.089737 26280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:07.089941 26170 server_base.cc:1034] running on GCE node
I20260619 14:52:07.090107 26170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:07.090396 26170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:07.091526 26170 hybrid_clock.cc:648] HybridClock initialized: now 1781880727091517 us; error 24 us; skew 500 ppm
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880727, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:07.092916 26170 init.cc:377] Logged in from keytab as oryx/127.19.34.125@KRBTEST.COM (short username oryx)
I20260619 14:52:07.093321 26170 webserver.cc:460] Webserver started at http://127.19.34.125:35259/ using document root <none> and password file <none>
I20260619 14:52:07.093488 26170 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:07.093538 26170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:07.094406 26170 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:52:07.095252 26287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:07.095468 26170 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:52:07.095534 26170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/wal
uuid: "5454034a11a7476699e9b65bb54329a3"
format_stamp: "Formatted at 2026-06-19 14:52:06 on dist-test-slave-3qdh"
server_key: "bf3cde73928da611d6eafb921b28bd0d"
server_key_iv: "85b613b9e955af758a2ba33e2eb3fcad"
server_key_version: "encryptionkey@0"
I20260619 14:52:07.095621 26170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:07.104863 26170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:07.106592 26170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:07.111327 26170 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.125:44365
I20260619 14:52:07.111325 26339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.125:44365 every 8 connection(s)
I20260619 14:52:07.111765 26170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-1/data/info.pb
I20260619 14:52:07.113116 26340 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:52:07.113835 26340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: Bootstrap starting.
I20260619 14:52:07.121210 26340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:52:07.121467 26340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: Bootstrap complete.
I20260619 14:52:07.121907 26340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: NON_VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: true } }
I20260619 14:52:07.121999 26340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5454034a11a7476699e9b65bb54329a3, State: Initialized, Role: LEARNER
I20260619 14:52:07.122097 26340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: NON_VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: true } }
I20260619 14:52:07.122390 26343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: NON_VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: true } } }
I20260619 14:52:07.122467 26340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:07.122531 26343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [sys.catalog]: This master's current role is: LEARNER
I20260619 14:52:07.125061 26354 catalog_manager.cc:1259] Loaded cluster ID: 09ee1b26ed4542908dcf941d1d1f6802
I20260619 14:52:07.125170 26354 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: loading cluster ID for follower catalog manager: success
I20260619 14:52:07.126663 26354 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: acquiring CA information for follower catalog manager: success
I20260619 14:52:07.127688 26354 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:52:07.169020 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.161850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39400 (local address 127.19.34.125:44365)
0619 14:52:07.162061 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.162071 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.162083 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:52:07.162118 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.162138 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.162170 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.162235 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.162938 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.163506 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.164586 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.164803 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.164972 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.166223 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.166226 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.166227 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.166245 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.167601 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.168011 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.168013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.168014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.168088 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.168398 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.168400 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.168401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.168553 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.168650 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.168817 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.168902 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for oryx/127.19.34.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.34.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.34.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.124.keytab.
Entry for principal oryx/127.19.34.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.124.keytab.
I20260619 14:52:07.193269 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.124:35023
--webserver_interface=127.19.34.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.124.keytab
--principal=oryx/127.19.34.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:44099,127.19.34.125:44365,127.19.34.124:35023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:07.291188 26358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:07.291383 26358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:07.291414 26358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:07.295311 26358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:07.295367 26358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:07.295387 26358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:07.295457 26358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:07.295488 26358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:52:07.300277 26358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:44099,127.19.34.125:44365,127.19.34.124:35023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.124.keytab
--principal=oryx/127.19.34.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.124:35023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.34.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:07.301141 26358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:07.301909 26358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:07.309243 26364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:07.309235 26363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:07.309233 26366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:07.309283 26358 server_base.cc:1034] running on GCE node
I20260619 14:52:07.309930 26358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:07.310773 26358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:07.311942 26358 hybrid_clock.cc:648] HybridClock initialized: now 1781880727311923 us; error 37 us; skew 500 ppm
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880727, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:07.314568 26358 init.cc:377] Logged in from keytab as oryx/127.19.34.124@KRBTEST.COM (short username oryx)
I20260619 14:52:07.315352 26358 webserver.cc:460] Webserver started at http://127.19.34.124:40065/ using document root <none> and password file <none>
I20260619 14:52:07.315809 26358 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:07.315861 26358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:07.316002 26358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:07.317730 26358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/data/instance:
uuid: "41b2bd4eba7e4448ad9b080bf12d71db"
format_stamp: "Formatted at 2026-06-19 14:52:07 on dist-test-slave-3qdh"
server_key: "b47ed56a7fcb222b9bfe67654be4cd5f"
server_key_iv: "267dba5dbccb8b9234dd4dc470414073"
server_key_version: "encryptionkey@0"
I20260619 14:52:07.318166 26358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/wal/instance:
uuid: "41b2bd4eba7e4448ad9b080bf12d71db"
format_stamp: "Formatted at 2026-06-19 14:52:07 on dist-test-slave-3qdh"
server_key: "b47ed56a7fcb222b9bfe67654be4cd5f"
server_key_iv: "267dba5dbccb8b9234dd4dc470414073"
server_key_version: "encryptionkey@0"
I20260619 14:52:07.321416 26358 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 14:52:07.323340 26373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:07.324481 26358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:07.324559 26358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/wal
uuid: "41b2bd4eba7e4448ad9b080bf12d71db"
format_stamp: "Formatted at 2026-06-19 14:52:07 on dist-test-slave-3qdh"
server_key: "b47ed56a7fcb222b9bfe67654be4cd5f"
server_key_iv: "267dba5dbccb8b9234dd4dc470414073"
server_key_version: "encryptionkey@0"
I20260619 14:52:07.324630 26358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:07.334502 26358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:07.336995 26358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:07.337121 26358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:07.343086 26358 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.124:35023
I20260619 14:52:07.343335 26425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.124:35023 every 8 connection(s)
I20260619 14:52:07.343983 26358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/data/info.pb
I20260619 14:52:07.346359 26426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:07.347934 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 26358
I20260619 14:52:07.349819 26426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880727, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.124@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.34.124@KRBTEST.COM
I20260619 14:52:07.361034 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.352046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51220 (local address 127.19.34.126:44099)
0619 14:52:07.352205 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.352209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.352420 (+ 211us) server_negotiation.cc:407] Connection header received
0619 14:52:07.352523 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.352526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.352575 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.352657 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.353222 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.353730 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.354733 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.355098 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.355222 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.357479 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.357484 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.357491 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.357509 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.359087 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.359545 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.359550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.359552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.359632 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.360010 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.360012 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.360013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.360234 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.360356 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.360796 (+ 440us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.360893 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:52:07.361689 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.351924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.0.0.1:51220)
0619 14:52:07.352282 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.352348 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.352447 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.352699 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.352702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.352961 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.353089 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.353106 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.353851 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.353854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.354615 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.354625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.355285 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.355290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.355413 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.355864 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.355887 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.357352 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.359212 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.359218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.359225 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.359454 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.359759 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.359762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.359764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.359919 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.360357 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.360363 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.360582 (+ 219us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.360739 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.360995 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":133,"threads_started":1}
I20260619 14:52:07.361688 26428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.349268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46916 (local address 127.19.34.124:35023)
0619 14:52:07.349669 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.349686 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.349720 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:52:07.350517 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.350553 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.350925 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.351225 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.352043 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.353189 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.354032 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.354526 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.354647 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.358027 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.358035 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.358047 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.358072 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.359658 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.360072 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.360078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.360079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.360145 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.360440 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.360442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.360443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.360633 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.360798 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.360981 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.361094 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"spinlock_wait_cycles":1408,"thread_start_us":108,"threads_started":1}
I20260619 14:52:07.362669 26426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880727, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.124@KRBTEST.COM for oryx/127.19.34.125@KRBTEST.COM
I20260619 14:52:07.369728 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.363046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:44365 (local address 127.0.0.1:39406)
0619 14:52:07.363344 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.363356 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.363489 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.363726 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.363728 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.363833 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.363903 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.363907 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.364633 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.364635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.365138 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.365145 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.365590 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.365593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.365661 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.365992 (+ 331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.366002 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.367017 (+ 1015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.368659 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.368662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.368663 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.368831 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.369050 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.369052 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.369053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.369120 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.369399 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.369401 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.369486 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.369550 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.369634 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242}
I20260619 14:52:07.369848 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.363210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39406 (local address 127.19.34.125:44365)
0619 14:52:07.363327 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.363331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.363428 (+ 97us) server_negotiation.cc:407] Connection header received
0619 14:52:07.363584 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.363586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.363639 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.363706 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.363989 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.364536 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.365244 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.365503 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.365578 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.367117 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.367119 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.367120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.367138 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.368572 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.368910 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.368912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.368913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.368977 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.369191 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.369193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.369194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.369333 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.369396 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.369604 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.369703 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:52:07.370524 26426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880727, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.124@KRBTEST.COM for oryx/127.19.34.124@KRBTEST.COM
I20260619 14:52:07.376965 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.370884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:35023 (local address 127.0.0.1:46926)
0619 14:52:07.371012 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.371023 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.371088 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.371274 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.371277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.371412 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.371457 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.371461 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.372373 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.372375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.372778 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.372783 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.373147 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.373149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.373222 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.373525 (+ 303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.373534 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.374418 (+ 884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.375728 (+ 1310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.375733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.375734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.375909 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.376186 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.376188 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.376190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.376253 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.376558 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.376562 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.376622 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.376715 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.376819 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 14:52:07.376983 26428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.370951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46926 (local address 127.19.34.124:35023)
0619 14:52:07.371033 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.371038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.371055 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:52:07.371160 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.371162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.371196 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.371266 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.371550 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.372250 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.372877 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.373060 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.373147 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.374517 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.374520 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.374522 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.374545 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.375586 (+ 1041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.375998 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.376004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.376006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.376085 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.376340 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.376345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.376347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.376471 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.376556 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.376751 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.376862 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":36}
I20260619 14:52:07.379082 26426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db: Bootstrap starting.
I20260619 14:52:07.380999 26426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:07.381608 26426 log.cc:824] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:07.383034 26426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db: No bootstrap required, opened a new log
I20260619 14:52:07.385550 26426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } }
I20260619 14:52:07.385738 26426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:07.385823 26426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41b2bd4eba7e4448ad9b080bf12d71db, State: Initialized, Role: FOLLOWER
I20260619 14:52:07.386274 26426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } }
I20260619 14:52:07.386626 26431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } } }
I20260619 14:52:07.386719 26431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:52:07.387207 26426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:52:07.391448 26442 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:52:07.391549 26442 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:52:07.394068 26358 master_runner.cc:428] Detected that this master 41b2bd4eba7e4448ad9b080bf12d71db is joining an existing cluster
I20260619 14:52:07.394132 26358 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.34.124:35023
I20260619 14:52:07.403681 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.396050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51224 (local address 127.19.34.126:44099)
0619 14:52:07.396183 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.396187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.396829 (+ 642us) server_negotiation.cc:407] Connection header received
0619 14:52:07.396955 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.396958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.396998 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.397067 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.397545 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.398184 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.399388 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.399685 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.399747 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.400898 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.400901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.400903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.400923 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.402156 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.402562 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.402567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.402569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.402652 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.402920 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.402924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.402926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.403102 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.403205 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.403467 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.403560 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 14:52:07.403697 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.396211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.0.0.1:51224)
0619 14:52:07.396764 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.396774 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.396876 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.397163 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.397166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.397347 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.397434 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.397440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.398298 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.398300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.399264 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.399273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.399894 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.399896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.400006 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.400479 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.400497 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.400767 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.402264 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.402269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.402271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.402464 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.402741 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.402745 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.402747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.402831 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.403194 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.403197 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.403312 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.403408 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.403532 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":54,"thread_start_us":101,"threads_started":1}
I20260619 14:52:07.408212 25897 heartbeater.cc:503] Master 127.19.34.126:44099 was elected leader, sending a full tablet report...
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880726, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.126@KRBTEST.COM for oryx/127.19.34.124@KRBTEST.COM
I20260619 14:52:07.415050 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 25768
I20260619 14:52:07.416731 26263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.406852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:35023 (local address 127.0.0.1:46940)
0619 14:52:07.406955 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.406970 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.407036 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.407354 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.407356 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.407483 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.407545 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.407549 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.408529 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.408535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.409226 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.409235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.409618 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.409621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.409700 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.410116 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.410126 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.411359 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.414589 (+ 3230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.414591 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.414605 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.414829 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.415562 (+ 733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.415568 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.415572 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.415679 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.416189 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.416193 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.416282 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.416373 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.416508 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 14:52:07.417311 26451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.406891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46940 (local address 127.19.34.124:35023)
0619 14:52:07.407122 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.407129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.407140 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:52:07.407215 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.407217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.407257 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.407335 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.407652 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.408442 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.409312 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.409522 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.409625 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.411485 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.411489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.411491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.411510 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.414485 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.414962 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.414964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.414966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.415046 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.415811 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.415817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.415821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.416070 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.416954 (+ 884us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.417023 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.417135 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":93,"threads_started":1}
I20260619 14:52:07.417579 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.396570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:35023 (local address 127.0.0.1:46938)
0619 14:52:07.397046 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.397056 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.397135 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.397383 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.397385 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.397543 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.397636 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.397642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.398496 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.398498 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.399450 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.399460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.400023 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.400026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.400133 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.413031 (+ 12898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.413044 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.413698 (+ 654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.416393 (+ 2695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.416395 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.416397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.416576 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.416772 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.416774 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.416775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.416828 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.417120 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.417122 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.417201 (+ 79us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.417302 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.417415 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":89,"thread_start_us":78,"threads_started":1}
I20260619 14:52:07.417968 26428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.396563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46938 (local address 127.19.34.124:35023)
0619 14:52:07.396643 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.396648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.397099 (+ 451us) server_negotiation.cc:407] Connection header received
0619 14:52:07.397217 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.397220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.397248 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.397286 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.397735 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.398388 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.399556 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.399829 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.399887 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.413814 (+ 13927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.413817 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.413819 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.413838 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.416281 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.416649 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.416652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.416653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.416706 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.416891 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.416893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.416895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.417032 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.417096 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.417743 (+ 647us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.417834 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":22,"spinlock_wait_cycles":1280}
I20260619 14:52:07.421614 25706 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.34.124:35023: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db"
member_type: NON_VOTER
last_known_addr {
host: "127.19.34.124"
port: 35023
}
attrs {
promote: true
}
}
dest_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc"
cas_config_opid_index: 5
I20260619 14:52:07.422242 25706 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: NON_VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: true } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: NON_VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: true } }
I20260619 14:52:07.422852 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:44803
--local_ip_for_outbound_sockets=127.19.34.65
--tserver_master_addrs=127.19.34.126:44099
--webserver_port=39301
--webserver_interface=127.19.34.65
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.65.keytab
--principal=oryx/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.34.126:44099,127.19.34.125:44365
--tserver_master_addrs=127.19.34.126:44099,127.19.34.125:44365,127.19.34.124:35023 with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:52:07.423030 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.396437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39422 (local address 127.19.34.125:44365)
0619 14:52:07.396539 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.396543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.396555 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:52:07.396661 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.396664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.396706 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.396780 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.397225 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.397891 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.399490 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.399775 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.399866 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.413237 (+ 13371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.413240 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.413242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.413265 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.414774 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.421437 (+ 6663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.421441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.421443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.421519 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.421845 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.421848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.421849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.422053 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.422141 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.422775 (+ 634us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.422885 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 14:52:07.423293 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.396092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:44365 (local address 127.0.0.1:39422)
0619 14:52:07.396460 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.396473 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.396547 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.396790 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.396792 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.396971 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.397119 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.397127 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.397992 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.397995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.399393 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.399400 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.399960 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.399962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.400028 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.412653 (+ 12625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.412663 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.413076 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.421098 (+ 8022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.421101 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.421103 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.421329 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.421622 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.421628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.421632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.421720 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.422185 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.422188 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.422270 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.422357 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.422476 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":68,"thread_start_us":89,"threads_started":1}
I20260619 14:52:07.423753 26265 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 41b2bd4eba7e4448ad9b080bf12d71db (127.19.34.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: NON_VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: true } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: NON_VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: true } } }
I20260619 14:52:07.425056 26265 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.34.124:35023
I20260619 14:52:07.425216 26265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: NON_VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: true } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: NON_VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: true } } }
I20260619 14:52:07.425287 26265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:07.425927 26358 master.cc:537] Master@127.19.34.124:35023 shutting down...
I20260619 14:52:07.426410 26400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:07.427736 26400 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 1 FOLLOWER]: Refusing update from remote peer 4e6af1f893a84c1d81d1a3c0ec3dc9bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 14:52:07.428251 26452 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: NON_VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:52:07.430663 25693 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc -> Peer 41b2bd4eba7e4448ad9b080bf12d71db (127.19.34.124:35023): Couldn't send request to peer 41b2bd4eba7e4448ad9b080bf12d71db. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 14:52:07.431285 26263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.425642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:44365 (local address 127.0.0.1:39432)
0619 14:52:07.426717 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.426730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.426809 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.427082 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.427085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.427123 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:07.427190 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.427195 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.428647 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.428650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.429796 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.429808 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.430879 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.430883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.430961 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.431001 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.431069 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.431151 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009}
I20260619 14:52:07.431339 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.425733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39432 (local address 127.19.34.125:44365)
0619 14:52:07.425838 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.425842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.426799 (+ 957us) server_negotiation.cc:407] Connection header received
0619 14:52:07.426901 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.426904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.426954 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.427042 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:07.427347 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.428528 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.429957 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.430735 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.430887 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.431060 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.431096 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.431188 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:52:07.431842 26314 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 1 LEARNER]: Refusing update from remote peer 4e6af1f893a84c1d81d1a3c0ec3dc9bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 14:52:07.432367 26452 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: NON_VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 14:52:07.432610 26358 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:07.432897 26358 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:07.433019 26358 tablet_replica.cc:330] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db: stopping tablet replica
I20260619 14:52:07.434396 26314 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 41b2bd4eba7e4448ad9b080bf12d71db (127.19.34.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: NON_VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: true } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: NON_VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: true } } }
I20260619 14:52:07.435964 26455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: NON_VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: true } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: NON_VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: true } } }
I20260619 14:52:07.436132 26455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [sys.catalog]: This master's current role is: LEARNER
I20260619 14:52:07.436575 26343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e6af1f893a84c1d81d1a3c0ec3dc9bc. Latest consensus state: current_term: 1 leader_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: NON_VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: true } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: NON_VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: true } } }
I20260619 14:52:07.436699 26343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [sys.catalog]: This master's current role is: LEARNER
I20260619 14:52:07.437309 26266 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc: attempting to promote NON_VOTER 5454034a11a7476699e9b65bb54329a3 to VOTER
I20260619 14:52:07.437659 26266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: NON_VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: true } }
I20260619 14:52:07.438863 26263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.438638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:35023 (local address 127.0.0.1:46956)
0619 14:52:07.438752 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.438806 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:35023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 14:52:07.439231 26314 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 1 LEARNER]: Refusing update from remote peer 4e6af1f893a84c1d81d1a3c0ec3dc9bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 14:52:07.439633 26456 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 14:52:07.439762 26263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.439564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:35023 (local address 127.0.0.1:46958)
0619 14:52:07.439671 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.439709 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:35023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 14:52:07.439921 25693 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc -> Peer 41b2bd4eba7e4448ad9b080bf12d71db (127.19.34.124:35023): Couldn't send request to peer 41b2bd4eba7e4448ad9b080bf12d71db. Status: Network error: Client connection negotiation failed: client connection to 127.19.34.124:35023: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 14:52:07.441064 26452 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5454034a11a7476699e9b65bb54329a3 (127.19.34.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: NON_VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: true } } }
I20260619 14:52:07.441418 26314 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5454034a11a7476699e9b65bb54329a3 (127.19.34.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: NON_VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: true } } }
I20260619 14:52:07.442771 26343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: NON_VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: true } } }
I20260619 14:52:07.442766 26265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: NON_VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: true } } }
I20260619 14:52:07.442865 26343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:52:07.442875 26265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:07.443028 26265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: NON_VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: true } } }
I20260619 14:52:07.443114 26265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:07.449870 26358 master.cc:552] Master@127.19.34.124:35023 shutdown complete.
I20260619 14:52:07.449918 26358 master_runner.cc:305] Clearing existing system tablet
I20260619 14:52:07.450160 26358 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:07.450220 26358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:07.451385 26358 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:52:07.452172 26458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:07.452416 26358 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:07.452528 26358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:52:07.452557 26358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/wal
uuid: "41b2bd4eba7e4448ad9b080bf12d71db"
format_stamp: "Formatted at 2026-06-19 14:52:07 on dist-test-slave-3qdh"
server_key: "b47ed56a7fcb222b9bfe67654be4cd5f"
server_key_iv: "267dba5dbccb8b9234dd4dc470414073"
server_key_version: "encryptionkey@0"
I20260619 14:52:07.454293 26358 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:07.456526 26358 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 14:52:07.456600 26358 log.cc:1196] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 14:52:07.456863 26358 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db: Deleting consensus metadata
I20260619 14:52:07.457005 26358 master_runner.cc:315] Copying system tablet from 127.19.34.126:44099
I20260619 14:52:07.457952 26358 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db: tablet copy: Beginning tablet copy session from remote peer at address 127.19.34.126:44099
I20260619 14:52:07.466081 26464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.458371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.0.0.1:51230)
0619 14:52:07.458648 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.458660 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.458775 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.458993 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.458995 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.459196 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.459313 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.459320 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.460265 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.460269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.461090 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.461099 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.461607 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.461612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.461737 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.462294 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.462308 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.462672 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.464523 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.464526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.464528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.464777 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.465073 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.465076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.465078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.465176 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.465563 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.465567 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.465661 (+ 94us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.465767 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.465888 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":71,"threads_started":1}
I20260619 14:52:07.466081 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.458431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51230 (local address 127.19.34.126:44099)
0619 14:52:07.458524 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.458528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.458753 (+ 225us) server_negotiation.cc:407] Connection header received
0619 14:52:07.458844 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.458847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.458889 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.458964 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.459422 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.460165 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.461202 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.461489 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.461577 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.462799 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.462803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.462806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.462829 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.464402 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.464876 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.464880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.464882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.464983 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.465270 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.465273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.465275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.465467 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.465560 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.465832 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.465934 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:52:07.466639 25726 tablet_copy_service.cc:140] P 4e6af1f893a84c1d81d1a3c0ec3dc9bc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 41b2bd4eba7e4448ad9b080bf12d71db ({username='oryx', principal='oryx/127.19.34.124@KRBTEST.COM'} at 127.0.0.1:51230)
I20260619 14:52:07.466718 25726 tablet_copy_service.cc:161] P 4e6af1f893a84c1d81d1a3c0ec3dc9bc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 41b2bd4eba7e4448ad9b080bf12d71db at {username='oryx', principal='oryx/127.19.34.124@KRBTEST.COM'} at 127.0.0.1:51230: session id = 41b2bd4eba7e4448ad9b080bf12d71db-00000000000000000000000000000000
I20260619 14:52:07.467607 25726 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 14:52:07.468783 26358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 14:52:07.470544 26358 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db: tablet copy: Starting download of 0 data blocks...
I20260619 14:52:07.470690 26358 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db: tablet copy: Starting download of 1 WAL segments...
I20260619 14:52:07.471982 26358 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 14:52:07.473423 25726 tablet_copy_service.cc:342] P 4e6af1f893a84c1d81d1a3c0ec3dc9bc: Request end of tablet copy session 41b2bd4eba7e4448ad9b080bf12d71db-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.34.124@KRBTEST.COM'} at 127.0.0.1:51230
I20260619 14:52:07.473492 25726 tablet_copy_service.cc:433] P 4e6af1f893a84c1d81d1a3c0ec3dc9bc: ending tablet copy session 41b2bd4eba7e4448ad9b080bf12d71db-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 41b2bd4eba7e4448ad9b080bf12d71db
I20260619 14:52:07.479821 26358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:07.481254 26465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:07.481292 26468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:07.481532 26466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:07.482012 26358 server_base.cc:1034] running on GCE node
I20260619 14:52:07.482184 26358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:07.482404 26358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:07.483534 26358 hybrid_clock.cc:648] HybridClock initialized: now 1781880727483523 us; error 24 us; skew 500 ppm
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880727, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:07.484911 26358 init.cc:377] Logged in from keytab as oryx/127.19.34.124@KRBTEST.COM (short username oryx)
I20260619 14:52:07.485337 26358 webserver.cc:460] Webserver started at http://127.19.34.124:40025/ using document root <none> and password file <none>
I20260619 14:52:07.485507 26358 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:07.485548 26358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:07.486665 26358 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:52:07.487619 26475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:07.487848 26358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:52:07.487912 26358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/wal
uuid: "41b2bd4eba7e4448ad9b080bf12d71db"
format_stamp: "Formatted at 2026-06-19 14:52:07 on dist-test-slave-3qdh"
server_key: "b47ed56a7fcb222b9bfe67654be4cd5f"
server_key_iv: "267dba5dbccb8b9234dd4dc470414073"
server_key_version: "encryptionkey@0"
I20260619 14:52:07.487977 26358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:07.499644 26358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:07.501644 26358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:07.506170 26358 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.124:35023
I20260619 14:52:07.506219 26527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.124:35023 every 8 connection(s)
I20260619 14:52:07.506659 26358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/master-2/data/info.pb
I20260619 14:52:07.508345 26528 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:52:07.509193 26528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db: Bootstrap starting.
I20260619 14:52:07.519842 26528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:52:07.520139 26528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db: Bootstrap complete.
I20260619 14:52:07.521173 26528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: NON_VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: true } }
I20260619 14:52:07.521301 26528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 41b2bd4eba7e4448ad9b080bf12d71db, State: Initialized, Role: LEARNER
I20260619 14:52:07.521435 26528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: NON_VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: true } }
I20260619 14:52:07.521736 26531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: NON_VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: true } } }
I20260619 14:52:07.521845 26528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:07.521914 26531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [sys.catalog]: This master's current role is: LEARNER
I20260619 14:52:07.524619 26542 catalog_manager.cc:1259] Loaded cluster ID: 09ee1b26ed4542908dcf941d1d1f6802
I20260619 14:52:07.524737 26542 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db: loading cluster ID for follower catalog manager: success
I20260619 14:52:07.526938 26542 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db: acquiring CA information for follower catalog manager: success
I20260619 14:52:07.528252 26542 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 14:52:07.549044 26453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:07.549284 26453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:07.549391 26453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:07.553098 26453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:07.553164 26453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:07.553236 26453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:52:07.558086 26453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.65.keytab
--principal=oryx/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:44803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=39301
--tserver_master_addrs=127.19.34.126:44099,127.19.34.125:44365,127.19.34.124:35023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:07.558941 26453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:07.559753 26453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:07.568706 26550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:07.568770 26548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:07.568725 26547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:07.569092 26453 server_base.cc:1034] running on GCE node
I20260619 14:52:07.569474 26453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:07.570024 26453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:07.571190 26453 hybrid_clock.cc:648] HybridClock initialized: now 1781880727571162 us; error 44 us; skew 500 ppm
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880727, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:07.573866 26453 init.cc:377] Logged in from keytab as oryx/127.19.34.65@KRBTEST.COM (short username oryx)
I20260619 14:52:07.574599 26453 webserver.cc:460] Webserver started at http://127.19.34.65:39301/ using document root <none> and password file <none>
I20260619 14:52:07.575100 26453 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:07.575169 26453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:07.578536 26453 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:52:07.580569 26557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:07.581744 26453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:07.581845 26453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "dc422cbf78144431af76287ded9cdf97"
format_stamp: "Formatted at 2026-06-19 14:52:06 on dist-test-slave-3qdh"
server_key: "e6a8db285ac2817ead64c08ccaa705fd"
server_key_iv: "8522fdc240c05212dde6fe281fe7a3b2"
server_key_version: "encryptionkey@0"
I20260619 14:52:07.582288 26453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:07.591792 26453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:07.592720 26031 heartbeater.cc:503] Master 127.19.34.126:44099 was elected leader, sending a full tablet report...
I20260619 14:52:07.594612 26453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:07.594789 26453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:07.595309 26453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:07.596172 26453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:07.596216 26453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:07.596273 26453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:07.596320 26453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:07.605031 26453 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:44803
I20260619 14:52:07.605072 26670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:44803 every 8 connection(s)
I20260619 14:52:07.605878 26453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:52:07.612258 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 26453
I20260619 14:52:07.612475 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 25902
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880727, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.125@KRBTEST.COM
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880727, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.124@KRBTEST.COM
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880727, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM
I20260619 14:52:07.621282 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.608402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:33433 (local address 127.19.34.125:44365)
0619 14:52:07.608541 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.608545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.609113 (+ 568us) server_negotiation.cc:407] Connection header received
0619 14:52:07.609977 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.609981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.610027 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.610085 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.611205 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.612002 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.613562 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.613881 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.613970 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.616987 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.616993 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.616997 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.617025 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.618578 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.619169 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.619175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.619178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.619265 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.619721 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.619727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.619729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.620005 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.620097 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.621013 (+ 916us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.621115 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:52:07.621712 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:39517
--local_ip_for_outbound_sockets=127.19.34.66
--tserver_master_addrs=127.19.34.126:44099
--webserver_port=35087
--webserver_interface=127.19.34.66
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.66.keytab
--principal=oryx/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.34.126:44099,127.19.34.125:44365
--tserver_master_addrs=127.19.34.126:44099,127.19.34.125:44365,127.19.34.124:35023 with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:52:07.621812 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.607429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:52635 (local address 127.19.34.126:44099)
0619 14:52:07.607560 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.607564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.608943 (+ 1379us) server_negotiation.cc:407] Connection header received
0619 14:52:07.609915 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.609918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.609969 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.610016 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.611203 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.612008 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.614414 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.614723 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.614819 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.618211 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.618215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.618220 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.618242 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.620039 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.620580 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.620583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.620585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.620665 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.620956 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.620959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.620961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.621157 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.621240 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.621564 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.621668 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:52:07.622150 26678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.609215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:53089 (local address 127.19.34.124:35023)
0619 14:52:07.609404 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.609407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.609418 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:52:07.609898 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.609900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.609956 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.610036 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.611198 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.611865 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.613907 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.614327 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.614433 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.618052 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.618058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.618064 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.618094 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.620119 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.620745 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.620751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.620755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.620852 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.621120 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.621126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.621130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.621347 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.621428 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.621794 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.621920 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":71,"threads_started":1}
I20260619 14:52:07.622762 26675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.607800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.65:52635)
0619 14:52:07.608793 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.608838 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.609716 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.610190 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.610196 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.610612 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.611075 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.611083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.612137 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.612140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.614291 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.614299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.615012 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.615018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.615115 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.615531 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.615551 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.618041 (+ 2490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.620160 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.620163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.620165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.620457 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.620771 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.620774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.620775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.620861 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.621293 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.621298 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.621397 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.621966 (+ 569us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.622071 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":223,"spinlock_wait_cycles":47104,"thread_start_us":560,"threads_started":1}
I20260619 14:52:07.622776 26676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.608288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:44365 (local address 127.19.34.65:33433)
0619 14:52:07.609044 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.609054 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.609716 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.610193 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.610196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.610615 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.611078 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.611083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.612137 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.612140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.613424 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.613432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.614117 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.614129 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.614359 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.614924 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.614946 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.616796 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.618699 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.618705 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.618719 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.619058 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.619371 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.619376 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.619379 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.619598 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.620152 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.620157 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.620502 (+ 345us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.620753 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.621791 (+ 1038us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":232,"spinlock_wait_cycles":21632,"thread_start_us":125,"threads_started":1}
I20260619 14:52:07.623267 26677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.609112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:35023 (local address 127.19.34.65:53089)
0619 14:52:07.609328 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.609340 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.609728 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.610188 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.610194 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.610553 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.611064 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.611075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.612028 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.612033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.613768 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.613782 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.614560 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.614566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.614688 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.615111 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.615132 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.617727 (+ 2595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.620240 (+ 2513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.620246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.620251 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.620619 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.620950 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.620953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.620954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.621030 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.621467 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.621470 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.621555 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.621637 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.622329 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":1114,"thread_start_us":85,"threads_started":1}
I20260619 14:52:07.624022 26672 heartbeater.cc:347] Connected to a master server at 127.19.34.126:44099
I20260619 14:52:07.624246 26673 heartbeater.cc:347] Connected to a master server at 127.19.34.125:44365
I20260619 14:52:07.624270 26672 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:07.624431 26673 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:07.624831 26673 heartbeater.cc:511] Master 127.19.34.125:44365 requested a full tablet report, sending...
I20260619 14:52:07.624835 26672 heartbeater.cc:511] Master 127.19.34.126:44099 requested a full tablet report, sending...
I20260619 14:52:07.625592 26671 heartbeater.cc:347] Connected to a master server at 127.19.34.124:35023
I20260619 14:52:07.625675 26671 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:07.625916 26671 heartbeater.cc:511] Master 127.19.34.124:35023 requested a full tablet report, sending...
I20260619 14:52:07.626037 25706 ts_manager.cc:194] Re-registered known tserver with Master: dc422cbf78144431af76287ded9cdf97 (127.19.34.65:44803)
I20260619 14:52:07.626348 26304 ts_manager.cc:194] Registered new tserver with Master: dc422cbf78144431af76287ded9cdf97 (127.19.34.65:44803)
I20260619 14:52:07.626708 25706 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:52635
I20260619 14:52:07.627555 26486 ts_manager.cc:194] Registered new tserver with Master: dc422cbf78144431af76287ded9cdf97 (127.19.34.65:44803)
W20260619 14:52:07.726812 26679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:07.727026 26679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:07.727065 26679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:07.731050 26679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:07.731102 26679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:07.731194 26679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:52:07.736035 26679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.66.keytab
--principal=oryx/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:39517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=35087
--tserver_master_addrs=127.19.34.126:44099,127.19.34.125:44365,127.19.34.124:35023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:07.736970 26679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:07.737910 26679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:07.747169 26685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:07.747182 26687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:07.747187 26684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:07.747382 26679 server_base.cc:1034] running on GCE node
I20260619 14:52:07.747725 26679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:07.748360 26679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:07.749536 26679 hybrid_clock.cc:648] HybridClock initialized: now 1781880727749516 us; error 36 us; skew 500 ppm
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880727, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:07.752038 26679 init.cc:377] Logged in from keytab as oryx/127.19.34.66@KRBTEST.COM (short username oryx)
I20260619 14:52:07.752879 26679 webserver.cc:460] Webserver started at http://127.19.34.66:35087/ using document root <none> and password file <none>
I20260619 14:52:07.753408 26679 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:07.753509 26679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:07.757026 26679 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:52:07.758893 26694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:07.759950 26679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:52:07.760071 26679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "bd8e60ecb8744ed2ab3e8a58713459dd"
format_stamp: "Formatted at 2026-06-19 14:52:06 on dist-test-slave-3qdh"
server_key: "8066407286d9d4427bb13a74ecaf4a12"
server_key_iv: "c4cc1b3635e8dcacecd6e0142e0676c1"
server_key_version: "encryptionkey@0"
I20260619 14:52:07.760596 26679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:07.778491 26679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:07.781108 26679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:07.781311 26679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:07.781805 26679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:07.782536 26679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:07.782598 26679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:07.782653 26679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:07.782697 26679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:07.792160 26679 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:39517
I20260619 14:52:07.792200 26807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:39517 every 8 connection(s)
I20260619 14:52:07.793213 26679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:52:07.799325 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 26679
I20260619 14:52:07.799500 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 26036
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880727, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.125@KRBTEST.COM
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880727, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM
I20260619 14:52:07.807317 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.795075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:39559 (local address 127.19.34.125:44365)
0619 14:52:07.795198 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.795203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.795864 (+ 661us) server_negotiation.cc:407] Connection header received
0619 14:52:07.797025 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.797028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.797074 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.797122 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.798419 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.799224 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.800539 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.800770 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.800879 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.803807 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.803811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.803814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.803838 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.805401 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.805904 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.805909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.805911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.805995 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.806331 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.806336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.806337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.806503 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.806599 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.807044 (+ 445us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.807152 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:52:07.807639 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.794859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:33895 (local address 127.19.34.126:44099)
0619 14:52:07.795038 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.795043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.795863 (+ 820us) server_negotiation.cc:407] Connection header received
0619 14:52:07.797012 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.797015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.797067 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.797162 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.798399 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.798910 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.800954 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.801368 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.801454 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.803835 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.803841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.803847 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.803879 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.805763 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.806269 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.806274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.806279 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.806356 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.806630 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.806636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.806640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.806888 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.806963 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.807329 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.807453 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 14:52:07.807963 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:33973
--local_ip_for_outbound_sockets=127.19.34.67
--tserver_master_addrs=127.19.34.126:44099
--webserver_port=44397
--webserver_interface=127.19.34.67
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.67.keytab
--principal=oryx/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:42451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.34.126:44099,127.19.34.125:44365
--tserver_master_addrs=127.19.34.126:44099,127.19.34.125:44365,127.19.34.124:35023 with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:52:07.808482 26812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.795292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.66:33895)
0619 14:52:07.795730 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.795771 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.796838 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.797306 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.797311 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.797713 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.798261 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.798274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.799041 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.799044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.800817 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.800830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.801611 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.801617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.801768 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.802241 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.802260 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.803674 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.805884 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.805890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.805896 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.806166 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.806455 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.806460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.806465 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.806544 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.807037 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.807043 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.807144 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.807953 (+ 809us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.808095 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":34,"spinlock_wait_cycles":12160,"thread_start_us":128,"threads_started":1}
I20260619 14:52:07.808485 26813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.795292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:44365 (local address 127.19.34.66:39559)
0619 14:52:07.795723 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.795771 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.796854 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.797305 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.797313 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.797725 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.798279 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.798288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.799346 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.799349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.800421 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.800437 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.800900 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.800905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.801075 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.801701 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.801715 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.803652 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.805538 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.805543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.805555 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.805802 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.806103 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.806106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.806108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.806245 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.806596 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.806603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.806834 (+ 231us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.807006 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.807433 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":66,"spinlock_wait_cycles":23296,"thread_start_us":137,"threads_started":1}
I20260619 14:52:07.809914 26810 heartbeater.cc:347] Connected to a master server at 127.19.34.125:44365
I20260619 14:52:07.809917 26809 heartbeater.cc:347] Connected to a master server at 127.19.34.126:44099
I20260619 14:52:07.810410 26810 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:07.810413 26809 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:07.810870 26809 heartbeater.cc:511] Master 127.19.34.126:44099 requested a full tablet report, sending...
I20260619 14:52:07.810875 26810 heartbeater.cc:511] Master 127.19.34.125:44365 requested a full tablet report, sending...
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880727, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.124@KRBTEST.COM
I20260619 14:52:07.811985 25706 ts_manager.cc:194] Re-registered known tserver with Master: bd8e60ecb8744ed2ab3e8a58713459dd (127.19.34.66:39517)
I20260619 14:52:07.812165 26304 ts_manager.cc:194] Registered new tserver with Master: bd8e60ecb8744ed2ab3e8a58713459dd (127.19.34.66:39517)
I20260619 14:52:07.812688 25706 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:33895
I20260619 14:52:07.815764 26678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.796596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:42269 (local address 127.19.34.124:35023)
0619 14:52:07.796750 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.796755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.796770 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:52:07.797025 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.797028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.797074 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.797122 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.798420 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.799233 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.809330 (+ 10097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.809640 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.809712 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.812127 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.812133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.812139 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.812184 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.814074 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.814618 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.814621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.814623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.814694 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.815013 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.815015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.815016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.815167 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.815242 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.815551 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.815648 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:52:07.816357 26814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.796242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:35023 (local address 127.19.34.66:42269)
0619 14:52:07.796530 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.796544 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.796854 (+ 310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.797305 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.797313 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.797723 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.798279 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.798288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.808434 (+ 10146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.808437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.809197 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.809206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.809865 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.809868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.809954 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.810402 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.810416 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.811972 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.814235 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.814241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.814244 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.814503 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.814792 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.814798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.814803 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.814918 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.815262 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.815268 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.815377 (+ 109us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.815499 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.815671 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":114,"spinlock_wait_cycles":12160,"thread_start_us":89,"threads_started":1}
I20260619 14:52:07.817072 26808 heartbeater.cc:347] Connected to a master server at 127.19.34.124:35023
I20260619 14:52:07.817169 26808 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:07.817343 26808 heartbeater.cc:511] Master 127.19.34.124:35023 requested a full tablet report, sending...
I20260619 14:52:07.817943 26486 ts_manager.cc:194] Registered new tserver with Master: bd8e60ecb8744ed2ab3e8a58713459dd (127.19.34.66:39517)
I20260619 14:52:07.822564 26678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.818843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.19.34.124:35023)
0619 14:52:07.818990 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.818996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.819012 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:52:07.819055 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.819061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.819141 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.819255 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:07.819494 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.820368 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.821389 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.822067 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.822172 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.822298 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.822394 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.822456 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:52:07.822599 26263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.818745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:35023 (local address 127.0.0.1:46962)
0619 14:52:07.818922 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.818933 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.818999 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.819264 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.819266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.819297 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:07.819374 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.819378 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.820467 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.820469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.821274 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.821283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.822213 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.822226 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.822299 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.822316 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.822371 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.822425 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260619 14:52:07.823235 26494 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 14:52:07.823555 26531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e6af1f893a84c1d81d1a3c0ec3dc9bc. Latest consensus state: current_term: 1 leader_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: NON_VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: true } } }
I20260619 14:52:07.823674 26531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [sys.catalog]: This master's current role is: LEARNER
W20260619 14:52:07.913511 26815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:07.913735 26815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:07.913796 26815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:07.917487 26815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:07.917552 26815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:07.917629 26815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:52:07.922224 26815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:42451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.67.keytab
--principal=oryx/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:33973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=44397
--tserver_master_addrs=127.19.34.126:44099,127.19.34.125:44365,127.19.34.124:35023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:07.923228 26815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:07.924158 26815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:07.932581 26820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:07.932610 26823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:07.932663 26821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:07.932786 26815 server_base.cc:1034] running on GCE node
I20260619 14:52:07.933161 26815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:07.933640 26815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:07.934824 26815 hybrid_clock.cc:648] HybridClock initialized: now 1781880727934802 us; error 47 us; skew 500 ppm
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880727, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:07.937497 26815 init.cc:377] Logged in from keytab as oryx/127.19.34.67@KRBTEST.COM (short username oryx)
I20260619 14:52:07.938256 26815 webserver.cc:460] Webserver started at http://127.19.34.67:44397/ using document root <none> and password file <none>
I20260619 14:52:07.938746 26815 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:07.938841 26815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:07.942323 26815 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 14:52:07.944425 26830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:07.945529 26815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:52:07.945642 26815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "8b22c40ce65d4e669cff4d97820eca21"
format_stamp: "Formatted at 2026-06-19 14:52:06 on dist-test-slave-3qdh"
server_key: "b90f3072e35578d2a4de0a7bf017f1e4"
server_key_iv: "e25b62983406fd733436eeba869e7bdf"
server_key_version: "encryptionkey@0"
I20260619 14:52:07.946094 26815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:07.958636 26815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:07.961082 26815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:07.961287 26815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:07.961783 26815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:07.962546 26815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:07.962641 26815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:07.962720 26815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:07.962750 26815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:07.972034 26815 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:33973
I20260619 14:52:07.972059 26943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:33973 every 8 connection(s)
I20260619 14:52:07.972894 26815 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:52:07.974339 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 26815
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880727, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880727, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.125@KRBTEST.COM
Jun 19 14:52:07 dist-test-slave-3qdh krb5kdc[25661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880727, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.124@KRBTEST.COM
I20260619 14:52:07.986860 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.976495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39438 (local address 127.19.34.125:44365)
0619 14:52:07.976601 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.976605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.978311 (+ 1706us) server_negotiation.cc:407] Connection header received
0619 14:52:07.978480 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.978482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.978525 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.978569 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.979439 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.980020 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.981047 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.981343 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.981409 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.983002 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.983005 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.983008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.983031 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.984683 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.985495 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.985499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.985501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.985579 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.985848 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.985851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.985853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.986058 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.986193 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.986569 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.986686 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:52:07.991385 26678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.976266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:34833 (local address 127.19.34.124:35023)
0619 14:52:07.976425 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.976431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.977442 (+ 1011us) server_negotiation.cc:407] Connection header received
0619 14:52:07.977535 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.977538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.977579 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.977656 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.979279 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.979966 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.981728 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.982012 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.982073 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.987697 (+ 5624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.987703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.987705 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.987733 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.989191 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.989748 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.989751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.989753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.989829 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.990261 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.990266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.990269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.990453 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.990529 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.991116 (+ 587us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.991236 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:52:07.992237 25745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.974481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:47983 (local address 127.19.34.126:44099)
0619 14:52:07.974597 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.974602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.975879 (+ 1277us) server_negotiation.cc:407] Connection header received
0619 14:52:07.977383 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.977386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.977465 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.977543 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.979279 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.979962 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.981728 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.982012 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.982073 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.986762 (+ 4689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.986765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.986767 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.986789 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.989104 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.991116 (+ 2012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.991119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.991121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.991200 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.991465 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.991468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.991470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.991655 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.991730 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.992016 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.992122 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:52:07.992774 26957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.976130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:35023 (local address 127.19.34.67:34833)
0619 14:52:07.977347 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.977359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.977452 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.977805 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.977813 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.978424 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.979120 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.979138 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.980120 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.980122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.981610 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.981625 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.982230 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.982233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.982460 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.983347 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.983365 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.986963 (+ 3598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.989315 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.989322 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.989335 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.989637 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.989942 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.989947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.989950 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.990148 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.990592 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.990596 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.990849 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.991521 (+ 672us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.991803 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1129,"mutex_wait_us":420,"spinlock_wait_cycles":32768,"thread_start_us":78,"threads_started":1}
I20260619 14:52:07.993700 26953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.976115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51234 (local address 127.19.34.126:44099)
0619 14:52:07.978078 (+ 1963us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.978084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.982042 (+ 3958us) server_negotiation.cc:407] Connection header received
0619 14:52:07.983087 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.983090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.983125 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.983206 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.984788 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.985658 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.986751 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.987077 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.987216 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.989514 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.989518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.989521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.989546 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.991204 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.991921 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.991925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.991928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.992029 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.992376 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.992381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.992386 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.992617 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.992728 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.993446 (+ 718us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.993551 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1897,"thread_start_us":74,"threads_started":1}
I20260619 14:52:07.993813 26944 heartbeater.cc:347] Connected to a master server at 127.19.34.124:35023
I20260619 14:52:07.993999 26944 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:07.994195 26954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.976519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:44365 (local address 127.19.34.67:55133)
0619 14:52:07.977575 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.977587 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.977659 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.977905 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.977908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.978436 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.979141 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.979153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.980122 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.980124 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.981580 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.981587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.982424 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.982427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.982534 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.983494 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.983510 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.987221 (+ 3711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.990743 (+ 3522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.990748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.990751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.993024 (+ 2273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.993294 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.993297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.993298 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.993372 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.993762 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.993764 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.993839 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.993940 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.994047 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"mutex_wait_us":370,"spinlock_wait_cycles":32256,"thread_start_us":102,"threads_started":1}
I20260619 14:52:07.994223 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.977028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:55133 (local address 127.19.34.125:44365)
0619 14:52:07.977237 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.977240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.977728 (+ 488us) server_negotiation.cc:407] Connection header received
0619 14:52:07.977765 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.977767 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.977804 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.977891 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.979286 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.979994 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.981697 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.982228 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.982316 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.987343 (+ 5027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.987348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.987353 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.987379 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.990660 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.993124 (+ 2464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.993127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.993129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.993208 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.993466 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.993469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.993471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.993651 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.993726 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.993996 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.994090 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":80,"threads_started":1}
I20260619 14:52:07.994407 26944 heartbeater.cc:511] Master 127.19.34.124:35023 requested a full tablet report, sending...
I20260619 14:52:07.994639 26952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.974800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.67:47983)
0619 14:52:07.975735 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:07.975785 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:07.977157 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:07.977805 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:07.977813 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:07.978400 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:07.979144 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.979149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.980130 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.980134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.981611 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:07.981619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.982230 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.982233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:07.982462 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.983366 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:07.983379 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:07.986573 (+ 3194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:07.990777 (+ 4204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.990780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.990781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.991021 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.991301 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:07.991303 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:07.991305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:07.991373 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:07.991785 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:07.991787 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:07.991858 (+ 71us) client_negotiation.cc:769] Sending connection context
0619 14:52:07.992690 (+ 832us) client_negotiation.cc:240] Negotiation successful
0619 14:52:07.992840 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":492,"spinlock_wait_cycles":6016,"thread_start_us":606,"threads_started":1}
I20260619 14:52:07.994771 26946 heartbeater.cc:347] Connected to a master server at 127.19.34.125:44365
I20260619 14:52:07.994832 26946 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:07.995030 26946 heartbeater.cc:511] Master 127.19.34.125:44365 requested a full tablet report, sending...
I20260619 14:52:07.995110 26945 heartbeater.cc:347] Connected to a master server at 127.19.34.126:44099
I20260619 14:52:07.995191 26945 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:07.995334 26945 heartbeater.cc:511] Master 127.19.34.126:44099 requested a full tablet report, sending...
I20260619 14:52:07.995810 26486 ts_manager.cc:194] Registered new tserver with Master: 8b22c40ce65d4e669cff4d97820eca21 (127.19.34.67:33973)
I20260619 14:52:07.995849 26304 ts_manager.cc:194] Registered new tserver with Master: 8b22c40ce65d4e669cff4d97820eca21 (127.19.34.67:33973)
I20260619 14:52:07.996402 25706 ts_manager.cc:194] Re-registered known tserver with Master: 8b22c40ce65d4e669cff4d97820eca21 (127.19.34.67:33973)
I20260619 14:52:07.997102 25706 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:47983
I20260619 14:52:07.997437 26958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:07.976827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46970 (local address 127.19.34.124:35023)
0619 14:52:07.977640 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:07.977643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:07.983259 (+ 5616us) server_negotiation.cc:407] Connection header received
0619 14:52:07.984069 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:07.984072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:07.984130 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:07.984205 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:07.985124 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.985943 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.988814 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:07.989154 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:07.989244 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:07.990948 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:07.990952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:07.990955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:07.990980 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:07.992635 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.993105 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.993109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.993113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.993188 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:07.993444 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:07.993446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:07.993447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:07.993609 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:07.993720 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:07.997144 (+ 3424us) server_negotiation.cc:299] Negotiation successful
0619 14:52:07.997241 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":77,"threads_started":1}
I20260619 14:52:07.996757 25701 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:52:08.000135 25701 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:52:08.002353 26456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: NON_VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: true } } }
I20260619 14:52:08.002468 26456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:08.005697 26962 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc: attempting to promote NON_VOTER 41b2bd4eba7e4448ad9b080bf12d71db to VOTER
I20260619 14:52:08.006022 26962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false } }
I20260619 14:52:08.007033 26314 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 1 FOLLOWER]: Refusing update from remote peer 4e6af1f893a84c1d81d1a3c0ec3dc9bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 14:52:08.007438 26963 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 14:52:08.007678 26502 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 1 LEARNER]: Refusing update from remote peer 4e6af1f893a84c1d81d1a3c0ec3dc9bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 14:52:08.008402 26963 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 14:52:08.009053 26963 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 41b2bd4eba7e4448ad9b080bf12d71db (127.19.34.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false } } }
I20260619 14:52:08.009932 26314 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 41b2bd4eba7e4448ad9b080bf12d71db (127.19.34.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false } } }
I20260619 14:52:08.010461 26963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false } } }
I20260619 14:52:08.010550 26963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:08.010639 26963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false } } }
I20260619 14:52:08.010691 26963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:08.011520 26972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false } } }
I20260619 14:52:08.011694 26972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:52:08.012166 26962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false } } }
I20260619 14:52:08.012270 26962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:08.012629 26502 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 41b2bd4eba7e4448ad9b080bf12d71db (127.19.34.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false } } }
I20260619 14:52:08.014354 26531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false } } }
I20260619 14:52:08.014501 26531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:52:08.018971 26973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:08.013648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:44803 (local address 127.0.0.1:48034)
0619 14:52:08.014059 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:08.014072 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:08.014152 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:08.015178 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:08.015181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:08.015198 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:08.015277 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:08.015285 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.016578 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.016581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:08.017728 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:08.017737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.018572 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.018576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:08.018654 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:08.018675 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:52:08.018749 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 14:52:08.018826 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":52,"thread_start_us":71,"threads_started":1}
I20260619 14:52:08.019052 26976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:08.014379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48034 (local address 127.19.34.65:44803)
0619 14:52:08.014682 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:08.014693 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:08.014712 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:52:08.014800 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:08.014806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:08.015029 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:08.015196 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:08.015411 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.016494 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.017854 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.018453 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.018573 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:08.018725 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:08.018787 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 14:52:08.018902 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":138,"threads_started":1}
I20260619 14:52:08.019155 26263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:08.013476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:39517 (local address 127.0.0.1:58270)
0619 14:52:08.013725 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:08.013741 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:08.013804 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:08.014904 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:08.014907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:08.014929 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:08.015043 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:08.015049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.016515 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.016518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:08.017728 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:08.017737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.018755 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.018758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:08.018892 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:08.018920 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 14:52:08.018967 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 14:52:08.019031 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":64}
I20260619 14:52:08.019302 26974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:08.013922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:33973 (local address 127.0.0.1:39310)
0619 14:52:08.014317 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:08.014328 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:08.014395 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:08.015842 (+ 1447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:08.015845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:08.015863 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:08.015939 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:08.015944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.017089 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.017095 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:08.017929 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:08.017938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.018931 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.018934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:08.019015 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:08.019032 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 14:52:08.019074 (+ 42us) client_negotiation.cc:240] Negotiation successful
0619 14:52:08.019142 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":29,"thread_start_us":75,"threads_started":1}
I20260619 14:52:08.019804 26977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:08.014719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.19.34.67:33973)
0619 14:52:08.015440 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:08.015454 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:08.015474 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:52:08.015531 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:08.015535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:08.015703 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:08.015838 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:08.016183 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.016988 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.018048 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.018784 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.019425 (+ 641us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:08.019547 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:08.019594 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 14:52:08.019672 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":119,"threads_started":1}
I20260619 14:52:08.019862 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:08.013987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58270 (local address 127.19.34.66:39517)
0619 14:52:08.014402 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:08.014413 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:08.014451 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:52:08.014535 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:08.014540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:08.014730 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:08.015201 (+ 471us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:08.015337 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.016400 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.017865 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.018593 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.019429 (+ 836us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:08.019547 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:08.019594 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 14:52:08.019680 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":128,"threads_started":1}
I20260619 14:52:08.022387 26742 tablet_service.cc:1467] Processing CreateTablet for tablet 37ebedc021c04046b3f450e19872e975 (DEFAULT_TABLE table=test-table [id=86241c8fc87d455e9d3707511815fdcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:08.022385 26605 tablet_service.cc:1467] Processing CreateTablet for tablet 37ebedc021c04046b3f450e19872e975 (DEFAULT_TABLE table=test-table [id=86241c8fc87d455e9d3707511815fdcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:08.023346 26605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ebedc021c04046b3f450e19872e975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:08.023347 26742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ebedc021c04046b3f450e19872e975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:08.026120 26878 tablet_service.cc:1467] Processing CreateTablet for tablet 37ebedc021c04046b3f450e19872e975 (DEFAULT_TABLE table=test-table [id=86241c8fc87d455e9d3707511815fdcd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:08.027070 26878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ebedc021c04046b3f450e19872e975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:08.030835 26978 tablet_bootstrap.cc:492] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd: Bootstrap starting.
I20260619 14:52:08.033013 26978 tablet_bootstrap.cc:654] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:08.033527 26979 tablet_bootstrap.cc:492] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21: Bootstrap starting.
I20260619 14:52:08.033803 26978 log.cc:824] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:08.034610 26980 tablet_bootstrap.cc:492] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97: Bootstrap starting.
I20260619 14:52:08.035445 26979 tablet_bootstrap.cc:654] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:08.035919 26978 tablet_bootstrap.cc:492] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd: No bootstrap required, opened a new log
I20260619 14:52:08.036100 26978 ts_tablet_manager.cc:1397] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:52:08.036204 26979 log.cc:824] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:08.036491 26980 tablet_bootstrap.cc:654] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:08.037186 26980 log.cc:824] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:08.038007 26979 tablet_bootstrap.cc:492] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21: No bootstrap required, opened a new log
I20260619 14:52:08.038168 26979 ts_tablet_manager.cc:1397] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:52:08.038722 26980 tablet_bootstrap.cc:492] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97: No bootstrap required, opened a new log
I20260619 14:52:08.038892 26980 ts_tablet_manager.cc:1397] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:52:08.040227 26978 raft_consensus.cc:348] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 } } peers { permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 } } peers { permanent_uuid: "8b22c40ce65d4e669cff4d97820eca21" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 33973 } }
I20260619 14:52:08.040520 26978 raft_consensus.cc:374] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:08.040620 26978 raft_consensus.cc:729] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd8e60ecb8744ed2ab3e8a58713459dd, State: Initialized, Role: FOLLOWER
I20260619 14:52:08.041043 26978 consensus_queue.cc:260] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 } } peers { permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 } } peers { permanent_uuid: "8b22c40ce65d4e669cff4d97820eca21" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 33973 } }
I20260619 14:52:08.042124 26978 ts_tablet_manager.cc:1428] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260619 14:52:08.042377 26979 raft_consensus.cc:348] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 } } peers { permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 } } peers { permanent_uuid: "8b22c40ce65d4e669cff4d97820eca21" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 33973 } }
I20260619 14:52:08.042613 26979 raft_consensus.cc:374] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:08.042677 26979 raft_consensus.cc:729] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b22c40ce65d4e669cff4d97820eca21, State: Initialized, Role: FOLLOWER
I20260619 14:52:08.043097 26979 consensus_queue.cc:260] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 } } peers { permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 } } peers { permanent_uuid: "8b22c40ce65d4e669cff4d97820eca21" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 33973 } }
I20260619 14:52:08.043880 26979 ts_tablet_manager.cc:1428] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:52:08.044301 26945 heartbeater.cc:503] Master 127.19.34.126:44099 was elected leader, sending a full tablet report...
W20260619 14:52:08.044416 26811 tablet.cc:2369] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:52:08.044767 26809 heartbeater.cc:503] Master 127.19.34.126:44099 was elected leader, sending a full tablet report...
I20260619 14:52:08.046315 26980 raft_consensus.cc:348] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 } } peers { permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 } } peers { permanent_uuid: "8b22c40ce65d4e669cff4d97820eca21" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 33973 } }
I20260619 14:52:08.047217 26980 raft_consensus.cc:374] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:08.047286 26980 raft_consensus.cc:729] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc422cbf78144431af76287ded9cdf97, State: Initialized, Role: FOLLOWER
I20260619 14:52:08.047670 26980 consensus_queue.cc:260] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 } } peers { permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 } } peers { permanent_uuid: "8b22c40ce65d4e669cff4d97820eca21" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 33973 } }
I20260619 14:52:08.048338 26672 heartbeater.cc:503] Master 127.19.34.126:44099 was elected leader, sending a full tablet report...
I20260619 14:52:08.048569 26980 ts_tablet_manager.cc:1428] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97: Time spent starting tablet: real 0.010s user 0.002s sys 0.004s
W20260619 14:52:08.107292 26674 tablet.cc:2369] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:52:08.116434 26985 raft_consensus.cc:482] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:52:08.116616 26985 raft_consensus.cc:504] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 } } peers { permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 } } peers { permanent_uuid: "8b22c40ce65d4e669cff4d97820eca21" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 33973 } }
I20260619 14:52:08.117703 26985 leader_election.cc:288] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd8e60ecb8744ed2ab3e8a58713459dd (127.19.34.66:39517), dc422cbf78144431af76287ded9cdf97 (127.19.34.65:44803)
I20260619 14:52:08.121776 26976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:08.118081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:49497 (local address 127.19.34.65:44803)
0619 14:52:08.118194 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:08.118199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:08.118214 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:52:08.118264 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:08.118267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:08.118316 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:08.118389 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:08.118627 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.119158 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.120557 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.121194 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.121309 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:08.121357 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:08.121597 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 14:52:08.121661 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:52:08.121800 26952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:08.117902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:44803 (local address 127.19.34.67:49497)
0619 14:52:08.118040 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:08.118054 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:08.118131 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:08.118419 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:08.118421 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:08.118432 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:08.118542 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:08.118547 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.119291 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.119295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:08.120433 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:08.120442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.121327 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.121333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:08.121426 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:08.121492 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 14:52:08.121581 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 14:52:08.121646 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":3}
I20260619 14:52:08.121848 26954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:08.117902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:39517 (local address 127.19.34.67:41977)
0619 14:52:08.118268 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:08.118278 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:08.118328 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:08.118517 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:08.118520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:08.118536 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:08.118646 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:08.118654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.119646 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.119649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:08.120632 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:08.120638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.121422 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.121425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:08.121553 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:08.121571 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:52:08.121634 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 14:52:08.121698 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":301}
I20260619 14:52:08.122465 26975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:08.118056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:41977 (local address 127.19.34.66:39517)
0619 14:52:08.118165 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:08.118170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:08.118318 (+ 148us) server_negotiation.cc:407] Connection header received
0619 14:52:08.118399 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:08.118401 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:08.118442 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:08.118505 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:08.118766 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.119535 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.120740 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.121303 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.121401 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:08.121448 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:08.122252 (+ 804us) server_negotiation.cc:299] Negotiation successful
0619 14:52:08.122324 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:52:08.122494 26625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ebedc021c04046b3f450e19872e975" candidate_uuid: "8b22c40ce65d4e669cff4d97820eca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc422cbf78144431af76287ded9cdf97" is_pre_election: true
I20260619 14:52:08.122756 26625 raft_consensus.cc:2454] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b22c40ce65d4e669cff4d97820eca21 in term 0.
I20260619 14:52:08.122906 26762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ebedc021c04046b3f450e19872e975" candidate_uuid: "8b22c40ce65d4e669cff4d97820eca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" is_pre_election: true
I20260619 14:52:08.123153 26762 raft_consensus.cc:2454] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b22c40ce65d4e669cff4d97820eca21 in term 0.
I20260619 14:52:08.123227 26832 leader_election.cc:302] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b22c40ce65d4e669cff4d97820eca21, dc422cbf78144431af76287ded9cdf97; no voters:
I20260619 14:52:08.123458 26985 raft_consensus.cc:2781] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:52:08.123545 26985 raft_consensus.cc:482] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:52:08.123588 26985 raft_consensus.cc:3037] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:08.124714 26985 raft_consensus.cc:504] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 } } peers { permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 } } peers { permanent_uuid: "8b22c40ce65d4e669cff4d97820eca21" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 33973 } }
I20260619 14:52:08.125052 26985 leader_election.cc:288] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 [CANDIDATE]: Term 1 election: Requested vote from peers bd8e60ecb8744ed2ab3e8a58713459dd (127.19.34.66:39517), dc422cbf78144431af76287ded9cdf97 (127.19.34.65:44803)
I20260619 14:52:08.125401 26762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ebedc021c04046b3f450e19872e975" candidate_uuid: "8b22c40ce65d4e669cff4d97820eca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd"
I20260619 14:52:08.125515 26762 raft_consensus.cc:3037] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:08.126538 26762 raft_consensus.cc:2454] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b22c40ce65d4e669cff4d97820eca21 in term 1.
I20260619 14:52:08.126775 26625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ebedc021c04046b3f450e19872e975" candidate_uuid: "8b22c40ce65d4e669cff4d97820eca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc422cbf78144431af76287ded9cdf97"
I20260619 14:52:08.126808 26834 leader_election.cc:302] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b22c40ce65d4e669cff4d97820eca21, bd8e60ecb8744ed2ab3e8a58713459dd; no voters:
I20260619 14:52:08.126892 26625 raft_consensus.cc:3037] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:08.126951 26985 raft_consensus.cc:2781] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:08.127168 26985 raft_consensus.cc:686] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 [term 1 LEADER]: Becoming Leader. State: Replica: 8b22c40ce65d4e669cff4d97820eca21, State: Running, Role: LEADER
I20260619 14:52:08.127485 26985 consensus_queue.cc:237] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 } } peers { permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 } } peers { permanent_uuid: "8b22c40ce65d4e669cff4d97820eca21" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 33973 } }
I20260619 14:52:08.128036 26625 raft_consensus.cc:2454] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b22c40ce65d4e669cff4d97820eca21 in term 1.
I20260619 14:52:08.130959 25701 catalog_manager.cc:5495] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b22c40ce65d4e669cff4d97820eca21 (127.19.34.67). New cstate: current_term: 1 leader_uuid: "8b22c40ce65d4e669cff4d97820eca21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b22c40ce65d4e669cff4d97820eca21" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 33973 } health_report { overall_health: HEALTHY } } }
I20260619 14:52:08.166743 26977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:08.163501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39318 (local address 127.19.34.67:33973)
0619 14:52:08.163614 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:08.163619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:08.163632 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:52:08.163765 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:08.163768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:08.163813 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:08.163890 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:52:08.164167 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.164815 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.165639 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.165847 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.165921 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:08.166174 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:52:08.166500 (+ 326us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:52:08.166575 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:08.166622 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 14:52:08.166662 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:52:08.171968 26625 raft_consensus.cc:1261] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97 [term 1 FOLLOWER]: Refusing update from remote peer 8b22c40ce65d4e669cff4d97820eca21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:08.171976 26762 raft_consensus.cc:1261] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd [term 1 FOLLOWER]: Refusing update from remote peer 8b22c40ce65d4e669cff4d97820eca21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:08.172628 26985 consensus_queue.cc:1035] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:08.172832 26987 consensus_queue.cc:1035] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:08.180575 26992 mvcc.cc:205] Tried to move back new op lower bound from 7298583462586306560 to 7298583462413172736. Current Snapshot: MvccSnapshot[applied={T|T < 7298583462586306560}]
I20260619 14:52:08.181962 26990 mvcc.cc:205] Tried to move back new op lower bound from 7298583462586306560 to 7298583462413172736. Current Snapshot: MvccSnapshot[applied={T|T < 7298583462586306560}]
I20260619 14:52:08.190932 25701 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:08.191075 25701 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:08.194361 25701 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e6af1f893a84c1d81d1a3c0ec3dc9bc: Sending DeleteTablet for 3 replicas of tablet 37ebedc021c04046b3f450e19872e975
I20260619 14:52:08.195266 26742 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ebedc021c04046b3f450e19872e975 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:08 UTC) from {username='oryx', principal='oryx/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:58270
I20260619 14:52:08.195708 26999 tablet_replica.cc:330] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd: stopping tablet replica
I20260619 14:52:08.195726 26878 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ebedc021c04046b3f450e19872e975 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:08 UTC) from {username='oryx', principal='oryx/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:39310
I20260619 14:52:08.195981 26999 raft_consensus.cc:2229] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:08.196192 27000 tablet_replica.cc:330] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21: stopping tablet replica
I20260619 14:52:08.196388 26999 raft_consensus.cc:2258] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:08.196321 26605 tablet_service.cc:1514] Processing DeleteTablet for tablet 37ebedc021c04046b3f450e19872e975 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:08 UTC) from {username='oryx', principal='oryx/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:48034
I20260619 14:52:08.196516 27000 raft_consensus.cc:2229] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:52:08.196831 27001 tablet_replica.cc:330] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97: stopping tablet replica
I20260619 14:52:08.196956 27000 raft_consensus.cc:2258] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:08.197067 27001 raft_consensus.cc:2229] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:08.197301 27001 raft_consensus.cc:2258] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:08.198558 27001 ts_tablet_manager.cc:1905] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:08.198794 27000 ts_tablet_manager.cc:1905] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:08.200268 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 25674
I20260619 14:52:08.202186 27001 ts_tablet_manager.cc:1918] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:08.202309 27000 ts_tablet_manager.cc:1918] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:08.202328 27001 log.cc:1196] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-0/wal/wals/37ebedc021c04046b3f450e19872e975
I20260619 14:52:08.202450 27000 log.cc:1196] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/37ebedc021c04046b3f450e19872e975
I20260619 14:52:08.202736 27001 ts_tablet_manager.cc:1939] T 37ebedc021c04046b3f450e19872e975 P dc422cbf78144431af76287ded9cdf97: Deleting consensus metadata
I20260619 14:52:08.202803 27000 ts_tablet_manager.cc:1939] T 37ebedc021c04046b3f450e19872e975 P 8b22c40ce65d4e669cff4d97820eca21: Deleting consensus metadata
I20260619 14:52:08.205320 26999 ts_tablet_manager.cc:1905] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 14:52:08.206943 26833 connection.cc:535] server connection from 127.0.0.1:39310 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 14:52:08.206966 26559 connection.cc:535] server connection from 127.0.0.1:48034 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:52:08.208724 26999 ts_tablet_manager.cc:1918] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:08.208860 26999 log.cc:1196] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-1/wal/wals/37ebedc021c04046b3f450e19872e975
I20260619 14:52:08.209204 26999 ts_tablet_manager.cc:1939] T 37ebedc021c04046b3f450e19872e975 P bd8e60ecb8744ed2ab3e8a58713459dd: Deleting consensus metadata
W20260619 14:52:08.210089 26696 connection.cc:441] server connection from 127.0.0.1:58270 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:58270 (request call id 1) could send its response
I20260619 14:52:08.214340 26958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:08.199005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.19.34.124:35023)
0619 14:52:08.199114 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:08.199118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:08.204934 (+ 5816us) server_negotiation.cc:407] Connection header received
0619 14:52:08.205129 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:08.205135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:08.205203 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:08.205258 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:08.206377 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.207254 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.208441 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.208743 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.208813 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:08.210970 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:08.210975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:08.210980 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:08.211007 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:08.212661 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:08.213086 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:08.213090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:08.213091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:08.213180 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:08.213446 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:08.213449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:08.213451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:08.213625 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:08.213735 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:08.214062 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 14:52:08.214182 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:52:08.218917 26678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:08.210616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46982 (local address 127.19.34.124:35023)
0619 14:52:08.210745 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:08.210752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:08.211033 (+ 281us) server_negotiation.cc:407] Connection header received
0619 14:52:08.211211 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:08.211214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:08.211262 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:08.211334 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:08.212110 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.212807 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.213875 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.214179 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.214249 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:08.215874 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:08.215878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:08.215880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:08.215902 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:08.217302 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:08.217786 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:08.217789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:08.217791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:08.217895 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:08.218151 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:08.218155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:08.218157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:08.218310 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:08.218427 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:08.218649 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 14:52:08.218742 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 14:52:08.221916 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:08.212611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39450 (local address 127.19.34.125:44365)
0619 14:52:08.212702 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:08.212706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:08.213038 (+ 332us) server_negotiation.cc:407] Connection header received
0619 14:52:08.213195 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:08.213198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:08.213238 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:08.213283 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:08.214130 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.215199 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.216689 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:08.217036 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:08.217118 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:08.218636 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:08.218639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:08.218642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:08.218666 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:08.220492 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:08.220835 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:08.220840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:08.220846 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:08.220922 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:08.221163 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:08.221167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:08.221170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:08.221382 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:08.221493 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:08.221654 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 14:52:08.221757 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 14:52:09.182955 27009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:09.182548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.65:35381)
0619 14:52:09.182804 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:09.182873 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":82,"threads_started":1}
W20260619 14:52:09.183241 26672 heartbeater.cc:644] Failed to heartbeat to 127.19.34.126:44099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
I20260619 14:52:09.186327 27010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:09.185869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.66:60029)
0619 14:52:09.186114 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:09.186238 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":89,"threads_started":1}
W20260619 14:52:09.186743 26809 heartbeater.cc:644] Failed to heartbeat to 127.19.34.126:44099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
I20260619 14:52:09.187541 27011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:09.187191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.67:34923)
0619 14:52:09.187398 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:09.187465 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":54,"threads_started":1}
W20260619 14:52:09.187811 26945 heartbeater.cc:644] Failed to heartbeat to 127.19.34.126:44099 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
I20260619 14:52:09.695680 27013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e6af1f893a84c1d81d1a3c0ec3dc9bc)
I20260619 14:52:09.695680 27012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e6af1f893a84c1d81d1a3c0ec3dc9bc)
I20260619 14:52:09.695911 27013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false } }
I20260619 14:52:09.695925 27012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false } }
I20260619 14:52:09.696861 27012 leader_election.cc:288] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e6af1f893a84c1d81d1a3c0ec3dc9bc (127.19.34.126:44099), 41b2bd4eba7e4448ad9b080bf12d71db (127.19.34.124:35023)
I20260619 14:52:09.697245 27013 leader_election.cc:288] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e6af1f893a84c1d81d1a3c0ec3dc9bc (127.19.34.126:44099), 5454034a11a7476699e9b65bb54329a3 (127.19.34.125:44365)
I20260619 14:52:09.697355 27014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:09.696969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.0.0.1:51688)
0619 14:52:09.697182 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:09.697258 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":71,"threads_started":1}
I20260619 14:52:09.697546 27015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:09.697023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.0.0.1:51694)
0619 14:52:09.697413 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:09.697475 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":67,"threads_started":1}
W20260619 14:52:09.697675 26291 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
W20260619 14:52:09.697708 26479 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
I20260619 14:52:09.698479 27014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:09.698276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.0.0.1:51708)
0619 14:52:09.698369 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:09.698419 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 14:52:09.698634 26291 leader_election.cc:334] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e6af1f893a84c1d81d1a3c0ec3dc9bc (127.19.34.126:44099): Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
I20260619 14:52:09.698900 27015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:09.698467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.0.0.1:51718)
0619 14:52:09.698588 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:09.698826 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260619 14:52:09.699021 26479 leader_election.cc:334] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e6af1f893a84c1d81d1a3c0ec3dc9bc (127.19.34.126:44099): Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
I20260619 14:52:09.701077 27019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:09.697552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46986 (local address 127.19.34.124:35023)
0619 14:52:09.697751 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:09.697755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:09.697970 (+ 215us) server_negotiation.cc:407] Connection header received
0619 14:52:09.698009 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:09.698012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:09.698049 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:09.698106 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:09.698407 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:09.699096 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:09.700079 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:09.700550 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:09.700656 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:09.700720 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:09.700876 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 14:52:09.700946 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":54,"threads_started":1}
I20260619 14:52:09.701124 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:09.697248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:35023 (local address 127.0.0.1:46986)
0619 14:52:09.697829 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:09.697839 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:09.697909 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:09.698187 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:09.698191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:09.698212 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:09.698311 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:09.698316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:09.699208 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:09.699210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:09.699949 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:09.699955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:09.700665 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:09.700669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:09.700762 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:09.700779 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 14:52:09.700862 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 14:52:09.700935 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":1,"thread_start_us":58,"threads_started":1}
I20260619 14:52:09.701114 27016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:09.697060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:44365 (local address 127.0.0.1:39464)
0619 14:52:09.697494 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:09.697509 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:09.697592 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:09.698028 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:09.698031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:09.698050 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:09.698155 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:09.698161 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:09.699162 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:09.699165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:09.699951 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:09.699959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:09.700704 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:09.700708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:09.700795 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:09.700823 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 14:52:09.700907 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 14:52:09.700986 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"spinlock_wait_cycles":768,"thread_start_us":41,"threads_started":1}
I20260619 14:52:09.701133 27018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:09.697410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39464 (local address 127.19.34.125:44365)
0619 14:52:09.697751 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:09.697755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:09.697766 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:52:09.697807 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:09.697810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:09.697856 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:09.697924 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:09.698250 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:09.699032 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:09.700093 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:09.700590 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:09.700705 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:09.700761 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:09.700921 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 14:52:09.701003 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":290,"thread_start_us":43,"threads_started":1}
I20260619 14:52:09.701634 26502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5454034a11a7476699e9b65bb54329a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" is_pre_election: true
I20260619 14:52:09.701757 26314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5454034a11a7476699e9b65bb54329a3" is_pre_election: true
I20260619 14:52:09.701824 26502 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5454034a11a7476699e9b65bb54329a3 in term 1.
I20260619 14:52:09.701931 26314 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41b2bd4eba7e4448ad9b080bf12d71db in term 1.
I20260619 14:52:09.702119 26291 leader_election.cc:302] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 41b2bd4eba7e4448ad9b080bf12d71db, 5454034a11a7476699e9b65bb54329a3; no voters: 4e6af1f893a84c1d81d1a3c0ec3dc9bc
I20260619 14:52:09.702323 27012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 14:52:09.702369 27012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e6af1f893a84c1d81d1a3c0ec3dc9bc)
I20260619 14:52:09.702394 27012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:52:09.702540 26476 leader_election.cc:302] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 41b2bd4eba7e4448ad9b080bf12d71db, 5454034a11a7476699e9b65bb54329a3; no voters: 4e6af1f893a84c1d81d1a3c0ec3dc9bc
I20260619 14:52:09.702850 27013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 14:52:09.702915 27013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e6af1f893a84c1d81d1a3c0ec3dc9bc)
I20260619 14:52:09.702963 27013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:52:09.703670 27012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false } }
I20260619 14:52:09.703996 27012 leader_election.cc:288] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [CANDIDATE]: Term 2 election: Requested vote from peers 4e6af1f893a84c1d81d1a3c0ec3dc9bc (127.19.34.126:44099), 41b2bd4eba7e4448ad9b080bf12d71db (127.19.34.124:35023)
I20260619 14:52:09.704147 27013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false } }
I20260619 14:52:09.704283 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:09.704079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.0.0.1:51724)
0619 14:52:09.704165 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:09.704209 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 14:52:09.704408 26502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5454034a11a7476699e9b65bb54329a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "41b2bd4eba7e4448ad9b080bf12d71db"
I20260619 14:52:09.704500 27013 leader_election.cc:288] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [CANDIDATE]: Term 2 election: Requested vote from peers 4e6af1f893a84c1d81d1a3c0ec3dc9bc (127.19.34.126:44099), 5454034a11a7476699e9b65bb54329a3 (127.19.34.125:44365)
I20260619 14:52:09.704538 26502 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5454034a11a7476699e9b65bb54329a3 in current term 2: Already voted for candidate 41b2bd4eba7e4448ad9b080bf12d71db in this term.
I20260619 14:52:09.704782 27016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:09.704577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.0.0.1:51726)
0619 14:52:09.704686 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:09.704732 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 14:52:09.705036 26314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5454034a11a7476699e9b65bb54329a3"
I20260619 14:52:09.705215 26314 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 41b2bd4eba7e4448ad9b080bf12d71db in current term 2: Already voted for candidate 5454034a11a7476699e9b65bb54329a3 in this term.
I20260619 14:52:09.705348 27017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:09.705119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.0.0.1:51742)
0619 14:52:09.705243 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:09.705291 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 14:52:09.705466 26291 leader_election.cc:334] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e6af1f893a84c1d81d1a3c0ec3dc9bc (127.19.34.126:44099): Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
I20260619 14:52:09.705530 26291 leader_election.cc:302] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5454034a11a7476699e9b65bb54329a3; no voters: 41b2bd4eba7e4448ad9b080bf12d71db, 4e6af1f893a84c1d81d1a3c0ec3dc9bc
I20260619 14:52:09.705549 27016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:09.705252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.0.0.1:51756)
0619 14:52:09.705443 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:09.705491 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260619 14:52:09.705652 26479 leader_election.cc:334] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e6af1f893a84c1d81d1a3c0ec3dc9bc (127.19.34.126:44099): Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
I20260619 14:52:09.705742 26479 leader_election.cc:302] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41b2bd4eba7e4448ad9b080bf12d71db; no voters: 4e6af1f893a84c1d81d1a3c0ec3dc9bc, 5454034a11a7476699e9b65bb54329a3
I20260619 14:52:09.705777 27012 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 14:52:09.705878 27013 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 14:52:10.184021 27022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:10.183673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.65:35961)
0619 14:52:10.183871 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:10.183939 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":60,"threads_started":1}
I20260619 14:52:10.188401 27023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:10.187194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.66:57027)
0619 14:52:10.188239 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:10.188316 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":980,"thread_start_us":73,"threads_started":1}
I20260619 14:52:10.188753 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:10.188376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.67:45431)
0619 14:52:10.188636 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:10.188687 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":76,"threads_started":1}
I20260619 14:52:11.185012 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.184666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.65:45117)
0619 14:52:11.184878 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.184929 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":65,"threads_started":1}
I20260619 14:52:11.189373 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.188934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.66:44463)
0619 14:52:11.189231 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.189286 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":87,"threads_started":1}
I20260619 14:52:11.189721 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.189366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.67:52089)
0619 14:52:11.189588 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.189648 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":84,"threads_started":1}
I20260619 14:52:11.471096 27028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:52:11.471289 27028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false } }
I20260619 14:52:11.471730 27028 leader_election.cc:288] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4e6af1f893a84c1d81d1a3c0ec3dc9bc (127.19.34.126:44099), 41b2bd4eba7e4448ad9b080bf12d71db (127.19.34.124:35023)
I20260619 14:52:11.472232 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.471898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.0.0.1:52040)
0619 14:52:11.472082 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.472152 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":58,"threads_started":1}
I20260619 14:52:11.472419 26502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5454034a11a7476699e9b65bb54329a3" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" is_pre_election: true
I20260619 14:52:11.472553 26502 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5454034a11a7476699e9b65bb54329a3 in term 2.
I20260619 14:52:11.473060 26291 leader_election.cc:302] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41b2bd4eba7e4448ad9b080bf12d71db, 5454034a11a7476699e9b65bb54329a3; no voters:
I20260619 14:52:11.473131 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.472924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.0.0.1:52050)
0619 14:52:11.473026 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.473072 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 14:52:11.473222 26291 leader_election.cc:334] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4e6af1f893a84c1d81d1a3c0ec3dc9bc (127.19.34.126:44099): Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
I20260619 14:52:11.473241 27028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 14:52:11.473299 27028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:52:11.473343 27028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 2 FOLLOWER]: Advancing to term 3
I20260619 14:52:11.474472 27028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false } }
I20260619 14:52:11.474790 27028 leader_election.cc:288] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [CANDIDATE]: Term 3 election: Requested vote from peers 4e6af1f893a84c1d81d1a3c0ec3dc9bc (127.19.34.126:44099), 41b2bd4eba7e4448ad9b080bf12d71db (127.19.34.124:35023)
I20260619 14:52:11.475127 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.474856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.0.0.1:52058)
0619 14:52:11.474999 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.475065 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 14:52:11.475335 26502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5454034a11a7476699e9b65bb54329a3" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "41b2bd4eba7e4448ad9b080bf12d71db"
I20260619 14:52:11.475463 26502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 2 FOLLOWER]: Advancing to term 3
I20260619 14:52:11.475852 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.475620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.0.0.1:52068)
0619 14:52:11.475728 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.475787 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260619 14:52:11.475955 26291 leader_election.cc:334] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4e6af1f893a84c1d81d1a3c0ec3dc9bc (127.19.34.126:44099): Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
I20260619 14:52:11.476770 26502 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5454034a11a7476699e9b65bb54329a3 in term 3.
I20260619 14:52:11.477030 26291 leader_election.cc:302] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 41b2bd4eba7e4448ad9b080bf12d71db, 5454034a11a7476699e9b65bb54329a3; no voters: 4e6af1f893a84c1d81d1a3c0ec3dc9bc
I20260619 14:52:11.477195 27028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 14:52:11.477370 27028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [term 3 LEADER]: Becoming Leader. State: Replica: 5454034a11a7476699e9b65bb54329a3, State: Running, Role: LEADER
I20260619 14:52:11.477604 27028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false } }
I20260619 14:52:11.478578 27031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5454034a11a7476699e9b65bb54329a3. Latest consensus state: current_term: 3 leader_uuid: "5454034a11a7476699e9b65bb54329a3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false } } }
I20260619 14:52:11.478700 27031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:11.478977 27033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:52:11.480997 27033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:52:11.481348 27033 catalog_manager.cc:1259] Loaded cluster ID: 09ee1b26ed4542908dcf941d1d1f6802
I20260619 14:52:11.481410 27033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:52:11.481915 27033 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:52:11.482291 27033 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: Loaded TSK: 0
I20260619 14:52:11.482904 27033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:52:11.491788 26304 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:52:11.494582 26304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:52:11.497212 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.497032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.0.0.1:52084)
0619 14:52:11.497122 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.497166 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260619 14:52:11.497439 26502 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [term 3 FOLLOWER]: Refusing update from remote peer 5454034a11a7476699e9b65bb54329a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 14:52:11.497843 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.497652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.0.0.1:52096)
0619 14:52:11.497756 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.497799 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 14:52:11.497864 27028 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260619 14:52:11.498020 26291 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 -> Peer 4e6af1f893a84c1d81d1a3c0ec3dc9bc (127.19.34.126:44099): Couldn't send request to peer 4e6af1f893a84c1d81d1a3c0ec3dc9bc. Status: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 14:52:11.499280 27035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5454034a11a7476699e9b65bb54329a3. Latest consensus state: current_term: 3 leader_uuid: "5454034a11a7476699e9b65bb54329a3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false } } }
I20260619 14:52:11.499385 27035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:52:11.500439 27028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "5454034a11a7476699e9b65bb54329a3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false } } }
I20260619 14:52:11.500537 27028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:11.500690 27036 mvcc.cc:205] Tried to move back new op lower bound from 7298583476209303552 to 7298583476134510592. Current Snapshot: MvccSnapshot[applied={T|T < 7298583476209303552}]
I20260619 14:52:11.500661 27035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "5454034a11a7476699e9b65bb54329a3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4e6af1f893a84c1d81d1a3c0ec3dc9bc" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44099 } } peers { permanent_uuid: "5454034a11a7476699e9b65bb54329a3" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 44365 } attrs { promote: false } } peers { permanent_uuid: "41b2bd4eba7e4448ad9b080bf12d71db" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 35023 } attrs { promote: false } } }
I20260619 14:52:11.501374 27035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41b2bd4eba7e4448ad9b080bf12d71db [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:52:11.509975 27029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.505790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:39517 (local address 127.0.0.1:58274)
0619 14:52:11.505889 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.505903 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:11.505991 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:11.506598 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:11.506602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:11.506635 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:11.506727 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:11.506732 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:11.507652 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:11.507655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:11.508829 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:11.508841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:11.509612 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:11.509616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:11.509700 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:11.509722 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:52:11.509793 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 14:52:11.509860 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 14:52:11.510318 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.505946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58274 (local address 127.19.34.66:39517)
0619 14:52:11.506304 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:11.506309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:11.506323 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:52:11.506419 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:11.506423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:11.506468 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:11.506831 (+ 363us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:11.506923 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:11.507543 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:11.508975 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:11.509474 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:11.509626 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:11.509708 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:11.510111 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 14:52:11.510204 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":61,"threads_started":1}
I20260619 14:52:11.510773 26742 tablet_service.cc:1467] Processing CreateTablet for tablet 7da39ee56f85478ebfaf911573377f14 (DEFAULT_TABLE table=test-table [id=c7776951b88f4f53aa31683ae1488a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:11.510927 27043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.506056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:33973 (local address 127.0.0.1:39332)
0619 14:52:11.506560 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.506570 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:11.506673 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:11.507212 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:11.507215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:11.507230 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:11.507297 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:11.507301 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:11.508128 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:11.508133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:11.509134 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:11.509143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:11.510565 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:11.510571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:11.510669 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:11.510689 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:52:11.510760 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 14:52:11.510823 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":20,"thread_start_us":98,"threads_started":1}
I20260619 14:52:11.510977 27045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.506787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39332 (local address 127.19.34.67:33973)
0619 14:52:11.506990 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:11.506994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:11.507006 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:52:11.507056 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:11.507058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:11.507102 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:11.507188 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:11.507444 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:11.508025 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:11.509264 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:11.510368 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:11.510453 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:11.510510 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:11.510791 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 14:52:11.510863 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":74,"threads_started":1}
I20260619 14:52:11.511057 26742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da39ee56f85478ebfaf911573377f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:11.511394 26878 tablet_service.cc:1467] Processing CreateTablet for tablet 7da39ee56f85478ebfaf911573377f14 (DEFAULT_TABLE table=test-table [id=c7776951b88f4f53aa31683ae1488a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:11.511588 26878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da39ee56f85478ebfaf911573377f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:11.513239 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.506368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:44803 (local address 127.0.0.1:48040)
0619 14:52:11.507411 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.507441 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:11.507523 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:11.507812 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:11.507815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:11.507847 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:11.507921 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:11.507926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:11.508885 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:11.508888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:11.510062 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:11.510074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:11.512841 (+ 2767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:11.512845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:11.512942 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:11.512964 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:52:11.513037 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 14:52:11.513100 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"mutex_wait_us":19,"thread_start_us":80,"threads_started":1}
I20260619 14:52:11.513638 27044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.506512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48040 (local address 127.19.34.65:44803)
0619 14:52:11.506764 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:11.506770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:11.507497 (+ 727us) server_negotiation.cc:407] Connection header received
0619 14:52:11.507598 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:11.507600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:11.507653 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:11.507723 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:11.508039 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:11.508784 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:11.510197 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:11.512596 (+ 2399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:11.512714 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:11.512769 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:11.513442 (+ 673us) server_negotiation.cc:299] Negotiation successful
0619 14:52:11.513521 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":63,"threads_started":1}
I20260619 14:52:11.513710 27047 tablet_bootstrap.cc:492] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21: Bootstrap starting.
I20260619 14:52:11.513856 27046 tablet_bootstrap.cc:492] T 7da39ee56f85478ebfaf911573377f14 P bd8e60ecb8744ed2ab3e8a58713459dd: Bootstrap starting.
I20260619 14:52:11.514349 26605 tablet_service.cc:1467] Processing CreateTablet for tablet 7da39ee56f85478ebfaf911573377f14 (DEFAULT_TABLE table=test-table [id=c7776951b88f4f53aa31683ae1488a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:11.514550 27047 tablet_bootstrap.cc:654] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:11.514602 26605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da39ee56f85478ebfaf911573377f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:11.515693 27047 tablet_bootstrap.cc:492] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21: No bootstrap required, opened a new log
I20260619 14:52:11.515790 27047 ts_tablet_manager.cc:1397] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:52:11.516204 27047 raft_consensus.cc:348] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 } } peers { permanent_uuid: "8b22c40ce65d4e669cff4d97820eca21" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 33973 } } peers { permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 } }
I20260619 14:52:11.516335 27047 raft_consensus.cc:374] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:11.516412 27047 raft_consensus.cc:729] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b22c40ce65d4e669cff4d97820eca21, State: Initialized, Role: FOLLOWER
I20260619 14:52:11.516561 27047 consensus_queue.cc:260] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 } } peers { permanent_uuid: "8b22c40ce65d4e669cff4d97820eca21" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 33973 } } peers { permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 } }
I20260619 14:52:11.516924 27046 tablet_bootstrap.cc:654] T 7da39ee56f85478ebfaf911573377f14 P bd8e60ecb8744ed2ab3e8a58713459dd: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:11.517061 27047 ts_tablet_manager.cc:1428] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 14:52:11.517351 27049 tablet_bootstrap.cc:492] T 7da39ee56f85478ebfaf911573377f14 P dc422cbf78144431af76287ded9cdf97: Bootstrap starting.
I20260619 14:52:11.518030 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.517812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.67:41509)
0619 14:52:11.517909 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.517971 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260619 14:52:11.518424 27046 tablet_bootstrap.cc:492] T 7da39ee56f85478ebfaf911573377f14 P bd8e60ecb8744ed2ab3e8a58713459dd: No bootstrap required, opened a new log
I20260619 14:52:11.518538 27046 ts_tablet_manager.cc:1397] T 7da39ee56f85478ebfaf911573377f14 P bd8e60ecb8744ed2ab3e8a58713459dd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 14:52:11.518899 27046 raft_consensus.cc:348] T 7da39ee56f85478ebfaf911573377f14 P bd8e60ecb8744ed2ab3e8a58713459dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 } } peers { permanent_uuid: "8b22c40ce65d4e669cff4d97820eca21" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 33973 } } peers { permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 } }
I20260619 14:52:11.518999 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.518404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.67:49527)
0619 14:52:11.518881 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.518944 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414}
I20260619 14:52:11.519003 27046 raft_consensus.cc:374] T 7da39ee56f85478ebfaf911573377f14 P bd8e60ecb8744ed2ab3e8a58713459dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:11.519040 27046 raft_consensus.cc:729] T 7da39ee56f85478ebfaf911573377f14 P bd8e60ecb8744ed2ab3e8a58713459dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd8e60ecb8744ed2ab3e8a58713459dd, State: Initialized, Role: FOLLOWER
I20260619 14:52:11.519240 27046 consensus_queue.cc:260] T 7da39ee56f85478ebfaf911573377f14 P bd8e60ecb8744ed2ab3e8a58713459dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 } } peers { permanent_uuid: "8b22c40ce65d4e669cff4d97820eca21" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 33973 } } peers { permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 } }
I20260619 14:52:11.519523 26946 heartbeater.cc:503] Master 127.19.34.125:44365 was elected leader, sending a full tablet report...
I20260619 14:52:11.519598 27046 ts_tablet_manager.cc:1428] T 7da39ee56f85478ebfaf911573377f14 P bd8e60ecb8744ed2ab3e8a58713459dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:52:11.520112 27049 tablet_bootstrap.cc:654] T 7da39ee56f85478ebfaf911573377f14 P dc422cbf78144431af76287ded9cdf97: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:11.520814 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.520255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.66:46447)
0619 14:52:11.520685 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.520739 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349}
I20260619 14:52:11.522082 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.521821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.66:44399)
0619 14:52:11.521974 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.522022 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 14:52:11.522626 27049 tablet_bootstrap.cc:492] T 7da39ee56f85478ebfaf911573377f14 P dc422cbf78144431af76287ded9cdf97: No bootstrap required, opened a new log
I20260619 14:52:11.522710 27049 ts_tablet_manager.cc:1397] T 7da39ee56f85478ebfaf911573377f14 P dc422cbf78144431af76287ded9cdf97: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260619 14:52:11.523133 27049 raft_consensus.cc:348] T 7da39ee56f85478ebfaf911573377f14 P dc422cbf78144431af76287ded9cdf97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 } } peers { permanent_uuid: "8b22c40ce65d4e669cff4d97820eca21" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 33973 } } peers { permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 } }
I20260619 14:52:11.523239 27049 raft_consensus.cc:374] T 7da39ee56f85478ebfaf911573377f14 P dc422cbf78144431af76287ded9cdf97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:11.523283 27049 raft_consensus.cc:729] T 7da39ee56f85478ebfaf911573377f14 P dc422cbf78144431af76287ded9cdf97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc422cbf78144431af76287ded9cdf97, State: Initialized, Role: FOLLOWER
I20260619 14:52:11.523398 27049 consensus_queue.cc:260] T 7da39ee56f85478ebfaf911573377f14 P dc422cbf78144431af76287ded9cdf97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 } } peers { permanent_uuid: "8b22c40ce65d4e669cff4d97820eca21" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 33973 } } peers { permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 } }
I20260619 14:52:11.523772 27049 ts_tablet_manager.cc:1428] T 7da39ee56f85478ebfaf911573377f14 P dc422cbf78144431af76287ded9cdf97: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:52:11.524240 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.524018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.65:33681)
0619 14:52:11.524125 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.524182 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 14:52:11.524338 26810 heartbeater.cc:503] Master 127.19.34.125:44365 was elected leader, sending a full tablet report...
I20260619 14:52:11.525285 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.525060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.65:47447)
0619 14:52:11.525166 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.525212 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 14:52:11.658529 27050 raft_consensus.cc:482] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:52:11.658703 27050 raft_consensus.cc:504] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 } } peers { permanent_uuid: "8b22c40ce65d4e669cff4d97820eca21" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 33973 } } peers { permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 } }
I20260619 14:52:11.659109 27050 leader_election.cc:288] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd8e60ecb8744ed2ab3e8a58713459dd (127.19.34.66:39517), dc422cbf78144431af76287ded9cdf97 (127.19.34.65:44803)
I20260619 14:52:11.659502 26762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da39ee56f85478ebfaf911573377f14" candidate_uuid: "8b22c40ce65d4e669cff4d97820eca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" is_pre_election: true
I20260619 14:52:11.659544 26625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da39ee56f85478ebfaf911573377f14" candidate_uuid: "8b22c40ce65d4e669cff4d97820eca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc422cbf78144431af76287ded9cdf97" is_pre_election: true
I20260619 14:52:11.659619 26762 raft_consensus.cc:2454] T 7da39ee56f85478ebfaf911573377f14 P bd8e60ecb8744ed2ab3e8a58713459dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b22c40ce65d4e669cff4d97820eca21 in term 0.
I20260619 14:52:11.659714 26625 raft_consensus.cc:2454] T 7da39ee56f85478ebfaf911573377f14 P dc422cbf78144431af76287ded9cdf97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b22c40ce65d4e669cff4d97820eca21 in term 0.
I20260619 14:52:11.659905 26834 leader_election.cc:302] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b22c40ce65d4e669cff4d97820eca21, bd8e60ecb8744ed2ab3e8a58713459dd; no voters:
I20260619 14:52:11.660063 27050 raft_consensus.cc:2781] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:52:11.660130 27050 raft_consensus.cc:482] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:52:11.660153 27050 raft_consensus.cc:3037] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:11.661082 27050 raft_consensus.cc:504] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 } } peers { permanent_uuid: "8b22c40ce65d4e669cff4d97820eca21" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 33973 } } peers { permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 } }
I20260619 14:52:11.661381 27050 leader_election.cc:288] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 [CANDIDATE]: Term 1 election: Requested vote from peers bd8e60ecb8744ed2ab3e8a58713459dd (127.19.34.66:39517), dc422cbf78144431af76287ded9cdf97 (127.19.34.65:44803)
I20260619 14:52:11.661715 26762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da39ee56f85478ebfaf911573377f14" candidate_uuid: "8b22c40ce65d4e669cff4d97820eca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd"
I20260619 14:52:11.661715 26625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da39ee56f85478ebfaf911573377f14" candidate_uuid: "8b22c40ce65d4e669cff4d97820eca21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc422cbf78144431af76287ded9cdf97"
I20260619 14:52:11.661834 26625 raft_consensus.cc:3037] T 7da39ee56f85478ebfaf911573377f14 P dc422cbf78144431af76287ded9cdf97 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:11.661834 26762 raft_consensus.cc:3037] T 7da39ee56f85478ebfaf911573377f14 P bd8e60ecb8744ed2ab3e8a58713459dd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:11.662901 26762 raft_consensus.cc:2454] T 7da39ee56f85478ebfaf911573377f14 P bd8e60ecb8744ed2ab3e8a58713459dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b22c40ce65d4e669cff4d97820eca21 in term 1.
I20260619 14:52:11.662901 26625 raft_consensus.cc:2454] T 7da39ee56f85478ebfaf911573377f14 P dc422cbf78144431af76287ded9cdf97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b22c40ce65d4e669cff4d97820eca21 in term 1.
I20260619 14:52:11.663172 26834 leader_election.cc:302] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b22c40ce65d4e669cff4d97820eca21, bd8e60ecb8744ed2ab3e8a58713459dd; no voters:
I20260619 14:52:11.663328 27050 raft_consensus.cc:2781] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:11.663543 27050 raft_consensus.cc:686] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 [term 1 LEADER]: Becoming Leader. State: Replica: 8b22c40ce65d4e669cff4d97820eca21, State: Running, Role: LEADER
I20260619 14:52:11.663729 27050 consensus_queue.cc:237] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 } } peers { permanent_uuid: "8b22c40ce65d4e669cff4d97820eca21" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 33973 } } peers { permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 } }
I20260619 14:52:11.664346 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.664085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.67:38971)
0619 14:52:11.664180 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.664255 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 14:52:11.665323 26304 catalog_manager.cc:5495] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b22c40ce65d4e669cff4d97820eca21 (127.19.34.67). New cstate: current_term: 1 leader_uuid: "8b22c40ce65d4e669cff4d97820eca21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b22c40ce65d4e669cff4d97820eca21" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 33973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 } health_report { overall_health: UNKNOWN } } }
I20260619 14:52:11.695034 27045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.692535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39348 (local address 127.19.34.67:33973)
0619 14:52:11.692647 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:11.692650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:11.692661 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:52:11.692730 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:11.692732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:11.692767 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:11.692827 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:52:11.693016 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:11.693482 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:11.694263 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:11.694457 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:11.694528 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:11.694717 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:52:11.694776 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:52:11.694827 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:11.694904 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 14:52:11.694944 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:52:11.696965 26625 raft_consensus.cc:1261] T 7da39ee56f85478ebfaf911573377f14 P dc422cbf78144431af76287ded9cdf97 [term 1 FOLLOWER]: Refusing update from remote peer 8b22c40ce65d4e669cff4d97820eca21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:11.697042 26762 raft_consensus.cc:1261] T 7da39ee56f85478ebfaf911573377f14 P bd8e60ecb8744ed2ab3e8a58713459dd [term 1 FOLLOWER]: Refusing update from remote peer 8b22c40ce65d4e669cff4d97820eca21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:11.697360 27055 consensus_queue.cc:1035] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd8e60ecb8744ed2ab3e8a58713459dd" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:11.697515 27050 consensus_queue.cc:1035] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc422cbf78144431af76287ded9cdf97" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:11.698773 26673 heartbeater.cc:503] Master 127.19.34.125:44365 was elected leader, sending a full tablet report...
I20260619 14:52:11.699491 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.699243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.65:45439)
0619 14:52:11.699377 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.699431 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 14:52:11.699565 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.699335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.66:59585)
0619 14:52:11.699465 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.699516 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260619 14:52:11.701752 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.701202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.67:60841)
0619 14:52:11.701648 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.701696 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386}
I20260619 14:52:11.702260 27060 mvcc.cc:205] Tried to move back new op lower bound from 7298583477026902016 to 7298583476895997952. Current Snapshot: MvccSnapshot[applied={T|T < 7298583477026902016 or (T in {7298583477026902016})}]
I20260619 14:52:11.702467 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.702178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.66:48397)
0619 14:52:11.702371 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.702415 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260619 14:52:11.705137 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.704911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.65:52155)
0619 14:52:11.705031 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.705069 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 14:52:11.706142 26302 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:11.706298 26302 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:11.707612 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.707444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44099 (local address 127.19.34.67:45673)
0619 14:52:11.707515 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.707561 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:44099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260619 14:52:11.709497 26302 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5454034a11a7476699e9b65bb54329a3: Sending DeleteTablet for 3 replicas of tablet 7da39ee56f85478ebfaf911573377f14
I20260619 14:52:11.710188 26742 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da39ee56f85478ebfaf911573377f14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:11 UTC) from {username='oryx', principal='oryx/127.19.34.125@KRBTEST.COM'} at 127.0.0.1:58274
I20260619 14:52:11.710192 26878 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da39ee56f85478ebfaf911573377f14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:11 UTC) from {username='oryx', principal='oryx/127.19.34.125@KRBTEST.COM'} at 127.0.0.1:39332
I20260619 14:52:11.710239 26605 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da39ee56f85478ebfaf911573377f14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:11 UTC) from {username='oryx', principal='oryx/127.19.34.125@KRBTEST.COM'} at 127.0.0.1:48040
I20260619 14:52:11.710480 27069 tablet_replica.cc:330] T 7da39ee56f85478ebfaf911573377f14 P dc422cbf78144431af76287ded9cdf97: stopping tablet replica
I20260619 14:52:11.710496 27068 tablet_replica.cc:330] T 7da39ee56f85478ebfaf911573377f14 P bd8e60ecb8744ed2ab3e8a58713459dd: stopping tablet replica
I20260619 14:52:11.710601 27068 raft_consensus.cc:2229] T 7da39ee56f85478ebfaf911573377f14 P bd8e60ecb8744ed2ab3e8a58713459dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:11.710600 27069 raft_consensus.cc:2229] T 7da39ee56f85478ebfaf911573377f14 P dc422cbf78144431af76287ded9cdf97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:11.710716 27068 raft_consensus.cc:2258] T 7da39ee56f85478ebfaf911573377f14 P bd8e60ecb8744ed2ab3e8a58713459dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:11.710743 27069 raft_consensus.cc:2258] T 7da39ee56f85478ebfaf911573377f14 P dc422cbf78144431af76287ded9cdf97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:11.711035 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 26453
I20260619 14:52:11.716531 27067 tablet_replica.cc:330] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21: stopping tablet replica
I20260619 14:52:11.716658 27067 raft_consensus.cc:2229] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:52:11.716845 27067 raft_consensus.cc:2258] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:11.717067 27068 ts_tablet_manager.cc:1905] T 7da39ee56f85478ebfaf911573377f14 P bd8e60ecb8744ed2ab3e8a58713459dd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 14:52:11.717211 27042 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:11.716998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:44803 (local address 127.0.0.1:48048)
0619 14:52:11.717092 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.717149 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.65:44803: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":26}
W20260619 14:52:11.717350 26289 catalog_manager.cc:4584] TS dc422cbf78144431af76287ded9cdf97 (127.19.34.65:44803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7da39ee56f85478ebfaf911573377f14: Network error: Client connection negotiation failed: client connection to 127.19.34.65:44803: connect: Connection reset by peer (error 104)
I20260619 14:52:11.717408 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 26679
I20260619 14:52:11.718145 27067 ts_tablet_manager.cc:1905] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:11.720896 27067 ts_tablet_manager.cc:1918] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:11.720973 27067 log.cc:1196] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/7da39ee56f85478ebfaf911573377f14
I20260619 14:52:11.721266 27067 ts_tablet_manager.cc:1939] T 7da39ee56f85478ebfaf911573377f14 P 8b22c40ce65d4e669cff4d97820eca21: Deleting consensus metadata
I20260619 14:52:11.721907 26289 catalog_manager.cc:4841] TS 8b22c40ce65d4e669cff4d97820eca21 (127.19.34.67:33973): tablet 7da39ee56f85478ebfaf911573377f14 (table test-table [id=c7776951b88f4f53aa31683ae1488a9d]) successfully deleted
I20260619 14:52:11.723997 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 26815
I20260619 14:52:11.724139 27042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:11.723921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:39517 (local address 127.0.0.1:58276)
0619 14:52:11.724010 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:11.724056 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.66:39517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 14:52:11.730489 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 26170
I20260619 14:52:11.735993 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 26358
2026-06-19T14:52:11Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6824 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:52:11 dist-test-slave-3qdh krb5kdc[27072](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:52:11 dist-test-slave-3qdh krb5kdc[27072](info): set up 2 sockets
Jun 19 14:52:11 dist-test-slave-3qdh krb5kdc[27072](info): commencing operation
krb5kdc: starting...
W20260619 14:52:12.728857 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.969s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:52:12 dist-test-slave-3qdh krb5kdc[27072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:52:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:52:12Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.126.keytab.
Entry for principal oryx/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.126.keytab.
I20260619 14:52:12.843212 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:43059
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.126.keytab
--principal=oryx/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:40509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:43059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:12.944449 27085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:12.944697 27085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:12.944758 27085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:12.948717 27085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:12.948786 27085 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 14:52:12.948817 27085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:12.948858 27085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:12.948889 27085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:12.948918 27085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:52:12.953779 27085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:40509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:43059
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.126.keytab
--principal=oryx/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:43059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:12.954648 27085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:12.955561 27085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:12.963150 27093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:12.963228 27090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:12.963254 27085 server_base.cc:1034] running on GCE node
W20260619 14:52:12.963157 27091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:12.964066 27085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:12.965145 27085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:12.966334 27085 hybrid_clock.cc:648] HybridClock initialized: now 1781880732966317 us; error 34 us; skew 500 ppm
Jun 19 14:52:12 dist-test-slave-3qdh krb5kdc[27072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880732, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:12.968925 27085 init.cc:377] Logged in from keytab as oryx/127.19.34.126@KRBTEST.COM (short username oryx)
I20260619 14:52:12.969687 27085 webserver.cc:460] Webserver started at http://127.19.34.126:44651/ using document root <none> and password file <none>
I20260619 14:52:12.970184 27085 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:12.970239 27085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:12.970389 27085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:12.972258 27085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "98e17d1a0e7848939ad4ac94f9912f16"
format_stamp: "Formatted at 2026-06-19 14:52:12 on dist-test-slave-3qdh"
server_key: "224912747a699312bd0e2758b97b8017"
server_key_iv: "9a91752411d024ab0e04d6e7aed3da55"
server_key_version: "encryptionkey@0"
I20260619 14:52:12.972824 27085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "98e17d1a0e7848939ad4ac94f9912f16"
format_stamp: "Formatted at 2026-06-19 14:52:12 on dist-test-slave-3qdh"
server_key: "224912747a699312bd0e2758b97b8017"
server_key_iv: "9a91752411d024ab0e04d6e7aed3da55"
server_key_version: "encryptionkey@0"
I20260619 14:52:12.976191 27085 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 14:52:12.978432 27100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:12.979619 27085 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:12.979743 27085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "98e17d1a0e7848939ad4ac94f9912f16"
format_stamp: "Formatted at 2026-06-19 14:52:12 on dist-test-slave-3qdh"
server_key: "224912747a699312bd0e2758b97b8017"
server_key_iv: "9a91752411d024ab0e04d6e7aed3da55"
server_key_version: "encryptionkey@0"
I20260619 14:52:12.979835 27085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:13.013679 27085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:13.016243 27085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:13.016459 27085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:13.023869 27085 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:43059
I20260619 14:52:13.023880 27162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:43059 every 8 connection(s)
I20260619 14:52:13.024734 27085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:52:13.027254 27163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:13.028097 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 27085
I20260619 14:52:13.028216 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:52:13.028494 19593 external_mini_cluster.cc:1428] Setting key 0863385e5043b93897240d729351aa3d
I20260619 14:52:13.032855 27163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16: Bootstrap starting.
I20260619 14:52:13.035277 27163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:13.036212 27163 log.cc:824] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:52:13 dist-test-slave-3qdh krb5kdc[27072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM
I20260619 14:52:13.038084 27163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16: No bootstrap required, opened a new log
I20260619 14:52:13.040509 27163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e17d1a0e7848939ad4ac94f9912f16" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43059 } }
I20260619 14:52:13.040728 27163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:13.040802 27163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98e17d1a0e7848939ad4ac94f9912f16, State: Initialized, Role: FOLLOWER
I20260619 14:52:13.041213 27163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e17d1a0e7848939ad4ac94f9912f16" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43059 } }
I20260619 14:52:13.041320 27163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:52:13.041388 27163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:52:13.041492 27163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:13.041708 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.029714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35934 (local address 127.19.34.126:43059)
0619 14:52:13.030071 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:13.030084 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:13.030115 (+ 31us) server_negotiation.cc:407] Connection header received
0619 14:52:13.030649 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:13.030671 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:13.030942 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:13.031176 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:13.031994 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.032972 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.034052 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.034448 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.034544 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.037280 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:13.037284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:13.037295 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:13.037317 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:13.039213 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:13.039589 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:13.039593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:13.039596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:13.039695 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:13.039936 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:13.039940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:13.039941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:13.040246 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:13.040468 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:13.040709 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 14:52:13.040971 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":122,"threads_started":1}
I20260619 14:52:13.042332 27163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e17d1a0e7848939ad4ac94f9912f16" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43059 } }
I20260619 14:52:13.042673 27163 leader_election.cc:302] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98e17d1a0e7848939ad4ac94f9912f16; no voters:
I20260619 14:52:13.042960 27163 leader_election.cc:288] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:52:13.043082 27168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:13.043300 27168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16 [term 1 LEADER]: Becoming Leader. State: Replica: 98e17d1a0e7848939ad4ac94f9912f16, State: Running, Role: LEADER
I20260619 14:52:13.043622 27168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e17d1a0e7848939ad4ac94f9912f16" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43059 } }
I20260619 14:52:13.044015 27163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:13.045315 27169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98e17d1a0e7848939ad4ac94f9912f16. Latest consensus state: current_term: 1 leader_uuid: "98e17d1a0e7848939ad4ac94f9912f16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e17d1a0e7848939ad4ac94f9912f16" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43059 } } }
I20260619 14:52:13.045364 27170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98e17d1a0e7848939ad4ac94f9912f16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98e17d1a0e7848939ad4ac94f9912f16" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43059 } } }
I20260619 14:52:13.045550 27170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:13.045823 27169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:13.045892 27180 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:52:13.048756 27180 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:52:13.053122 27180 catalog_manager.cc:1347] Generated new cluster ID: 5b8a74f806c74c19a74845a0bffc28ca
I20260619 14:52:13.053215 27180 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:52:13.066471 27180 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:52:13.067384 27180 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:52:13.081338 27180 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16: Generated new TSK 0
I20260619 14:52:13.081979 27180 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.34.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.34.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.34.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.65.keytab.
Entry for principal oryx/127.19.34.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.65.keytab.
I20260619 14:52:13.115144 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:43059
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.65.keytab
--principal=oryx/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:40509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:13.223392 27189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:13.223645 27189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:13.223717 27189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:13.227490 27189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:13.227568 27189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:13.227638 27189 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:52:13.227716 27189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:52:13.232753 27189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:40509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.65.keytab
--principal=oryx/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.34.126:43059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:13.233680 27189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:13.234617 27189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:13.243860 27194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:13.243901 27195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:13.243960 27189 server_base.cc:1034] running on GCE node
W20260619 14:52:13.243880 27197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:13.244554 27189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:13.245143 27189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:13.246364 27189 hybrid_clock.cc:648] HybridClock initialized: now 1781880733246340 us; error 40 us; skew 500 ppm
Jun 19 14:52:13 dist-test-slave-3qdh krb5kdc[27072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880733, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:13.249125 27189 init.cc:377] Logged in from keytab as oryx/127.19.34.65@KRBTEST.COM (short username oryx)
I20260619 14:52:13.249888 27189 webserver.cc:460] Webserver started at http://127.19.34.65:46305/ using document root <none> and password file <none>
I20260619 14:52:13.250428 27189 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:13.250514 27189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:13.250773 27189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:13.252723 27189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "1cf2f87cb790496882ab5c0299489e92"
format_stamp: "Formatted at 2026-06-19 14:52:13 on dist-test-slave-3qdh"
server_key: "2769a7f37f0681af4535c7c12bc16d32"
server_key_iv: "6d9d1a64f892ef1363e543e243daec2c"
server_key_version: "encryptionkey@0"
I20260619 14:52:13.253268 27189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "1cf2f87cb790496882ab5c0299489e92"
format_stamp: "Formatted at 2026-06-19 14:52:13 on dist-test-slave-3qdh"
server_key: "2769a7f37f0681af4535c7c12bc16d32"
server_key_iv: "6d9d1a64f892ef1363e543e243daec2c"
server_key_version: "encryptionkey@0"
I20260619 14:52:13.256649 27189 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:52:13.258798 27204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:13.259841 27189 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:13.259961 27189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "1cf2f87cb790496882ab5c0299489e92"
format_stamp: "Formatted at 2026-06-19 14:52:13 on dist-test-slave-3qdh"
server_key: "2769a7f37f0681af4535c7c12bc16d32"
server_key_iv: "6d9d1a64f892ef1363e543e243daec2c"
server_key_version: "encryptionkey@0"
I20260619 14:52:13.260058 27189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:13.282475 27189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:13.285465 27189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:13.285801 27189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:13.287318 27189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:13.287372 27189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:13.287441 27189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:13.287477 27189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:52:13 dist-test-slave-3qdh krb5kdc[27072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880733, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM
I20260619 14:52:13.298803 27189 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:34119
I20260619 14:52:13.298870 27319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:34119 every 8 connection(s)
I20260619 14:52:13.299808 27189 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:52:13.300889 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.288146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:37767 (local address 127.19.34.126:43059)
0619 14:52:13.288265 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:13.288268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:13.289311 (+ 1043us) server_negotiation.cc:407] Connection header received
0619 14:52:13.290256 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:13.290262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:13.290332 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:13.290440 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:13.291877 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.292796 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.293944 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.294229 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.294336 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.297019 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:13.297023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:13.297028 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:13.297056 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:13.298797 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:13.299306 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:13.299309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:13.299311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:13.299413 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:13.299726 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:13.299732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:13.299737 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:13.299967 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:13.300090 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:13.300611 (+ 521us) server_negotiation.cc:299] Negotiation successful
0619 14:52:13.300722 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:52:13.301115 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 27189
I20260619 14:52:13.301230 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:52:13.301527 19593 external_mini_cluster.cc:1428] Setting key 0d438dd9552cab856f1fedeb01eb4718
I20260619 14:52:13.301648 27213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.288715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43059 (local address 127.19.34.65:37767)
0619 14:52:13.289178 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:13.289213 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:13.290023 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:13.290643 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:13.290653 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:13.291105 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:13.291686 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.291704 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.292936 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.292939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.293810 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.293819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.294386 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.294390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.294564 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.295106 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:13.295132 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:13.296865 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:13.298937 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:13.298944 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:13.298957 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:13.299204 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:13.299510 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:13.299512 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:13.299513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:13.299633 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:13.300068 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:13.300072 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:13.300325 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 14:52:13.300501 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 14:52:13.300769 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":145,"threads_started":1}
I20260619 14:52:13.303035 27320 heartbeater.cc:347] Connected to a master server at 127.19.34.126:43059
I20260619 14:52:13.304878 27320 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:13.305330 27320 heartbeater.cc:511] Master 127.19.34.126:43059 requested a full tablet report, sending...
I20260619 14:52:13.306702 27117 ts_manager.cc:194] Registered new tserver with Master: 1cf2f87cb790496882ab5c0299489e92 (127.19.34.65:34119)
I20260619 14:52:13.307801 27117 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:37767
WARNING: no policy specified for oryx/127.19.34.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.34.66@KRBTEST.COM" created.
I20260619 14:52:13.315371 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.304845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43059 (local address 127.19.34.65:36769)
0619 14:52:13.308490 (+ 3645us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:13.308507 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:13.308613 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:13.308867 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:13.308869 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:13.309019 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:13.309103 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.309109 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.310160 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.310162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.310700 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.310707 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.311260 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.311262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.311323 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.311833 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:13.311846 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:13.312246 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:13.313942 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:13.313947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:13.313949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:13.314205 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:13.314473 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:13.314478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:13.314483 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:13.314582 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:13.314903 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:13.314909 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:13.315015 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 14:52:13.315130 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 14:52:13.315238 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3569,"thread_start_us":102,"threads_started":1}
I20260619 14:52:13.315533 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.304897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:36769 (local address 127.19.34.126:43059)
0619 14:52:13.305003 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:13.305007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:13.308589 (+ 3582us) server_negotiation.cc:407] Connection header received
0619 14:52:13.308696 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:13.308701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:13.308759 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:13.308851 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:13.309215 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.310032 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.310813 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.311157 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.311264 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.312389 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:13.312392 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:13.312395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:13.312418 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:13.313826 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:13.314315 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:13.314317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:13.314318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:13.314387 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:13.314676 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:13.314678 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:13.314679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:13.314815 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:13.314884 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:13.315206 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 14:52:13.315329 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.34.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.66.keytab.
Entry for principal oryx/127.19.34.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.66.keytab.
I20260619 14:52:13.327106 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:43059
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.66.keytab
--principal=oryx/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:40509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:13.431409 27329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:13.431681 27329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:13.431752 27329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:13.435565 27329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:13.435653 27329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:13.435720 27329 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:52:13.435784 27329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:52:13.440917 27329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:40509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.66.keytab
--principal=oryx/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.34.126:43059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:13.441829 27329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:13.442791 27329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:13.452365 27335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:13.452358 27337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:13.452370 27334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:13.452579 27329 server_base.cc:1034] running on GCE node
I20260619 14:52:13.453015 27329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:13.453578 27329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:13.454763 27329 hybrid_clock.cc:648] HybridClock initialized: now 1781880733454745 us; error 30 us; skew 500 ppm
Jun 19 14:52:13 dist-test-slave-3qdh krb5kdc[27072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880733, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:13.457422 27329 init.cc:377] Logged in from keytab as oryx/127.19.34.66@KRBTEST.COM (short username oryx)
I20260619 14:52:13.458209 27329 webserver.cc:460] Webserver started at http://127.19.34.66:34623/ using document root <none> and password file <none>
I20260619 14:52:13.458746 27329 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:13.458830 27329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:13.459038 27329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:13.461392 27329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "01932012b1bd4b8cb18f21da85bf1871"
format_stamp: "Formatted at 2026-06-19 14:52:13 on dist-test-slave-3qdh"
server_key: "603b8edc71fe51ae71eb9f68faec563e"
server_key_iv: "97ec1ecb4da5480c9dd6423d5a2bf0fe"
server_key_version: "encryptionkey@0"
I20260619 14:52:13.461949 27329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "01932012b1bd4b8cb18f21da85bf1871"
format_stamp: "Formatted at 2026-06-19 14:52:13 on dist-test-slave-3qdh"
server_key: "603b8edc71fe51ae71eb9f68faec563e"
server_key_iv: "97ec1ecb4da5480c9dd6423d5a2bf0fe"
server_key_version: "encryptionkey@0"
I20260619 14:52:13.465437 27329 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:52:13.467587 27344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:13.468804 27329 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:52:13.468928 27329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "01932012b1bd4b8cb18f21da85bf1871"
format_stamp: "Formatted at 2026-06-19 14:52:13 on dist-test-slave-3qdh"
server_key: "603b8edc71fe51ae71eb9f68faec563e"
server_key_iv: "97ec1ecb4da5480c9dd6423d5a2bf0fe"
server_key_version: "encryptionkey@0"
I20260619 14:52:13.469043 27329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:13.480693 27329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:13.483646 27329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:13.483831 27329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:13.485229 27329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:13.485299 27329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:13.485368 27329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:13.485422 27329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:52:13 dist-test-slave-3qdh krb5kdc[27072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880733, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM
I20260619 14:52:13.495822 27329 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:43377
I20260619 14:52:13.496455 27459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:43377 every 8 connection(s)
I20260619 14:52:13.496789 27329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:52:13.498417 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.485795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:55979 (local address 127.19.34.126:43059)
0619 14:52:13.485929 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:13.485934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:13.486783 (+ 849us) server_negotiation.cc:407] Connection header received
0619 14:52:13.487705 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:13.487708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:13.487765 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:13.487868 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:13.489218 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.489911 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.491113 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.491454 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.491570 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.494705 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:13.494710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:13.494713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:13.494742 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:13.496351 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:13.496940 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:13.496942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:13.496944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:13.497014 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:13.497403 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:13.497408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:13.497410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:13.497640 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:13.497742 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:13.498211 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 14:52:13.498303 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:52:13.499086 27353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.486208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43059 (local address 127.19.34.66:55979)
0619 14:52:13.486620 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:13.486664 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:13.487503 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:13.488172 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:13.488182 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:13.488638 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:13.489068 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.489081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.490036 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.490040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.490988 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.490997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.491599 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.491603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.491769 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.492358 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:13.492377 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:13.494498 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:13.496482 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:13.496489 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:13.496502 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:13.496845 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:13.497112 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:13.497116 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:13.497118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:13.497285 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:13.497729 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:13.497734 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:13.498011 (+ 277us) client_negotiation.cc:769] Sending connection context
0619 14:52:13.498158 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 14:52:13.498366 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":114,"threads_started":1}
I20260619 14:52:13.500243 27460 heartbeater.cc:347] Connected to a master server at 127.19.34.126:43059
I20260619 14:52:13.500404 27460 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:13.500892 27460 heartbeater.cc:511] Master 127.19.34.126:43059 requested a full tablet report, sending...
I20260619 14:52:13.501955 27115 ts_manager.cc:194] Registered new tserver with Master: 01932012b1bd4b8cb18f21da85bf1871 (127.19.34.66:43377)
I20260619 14:52:13.502262 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 27329
I20260619 14:52:13.502386 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:52:13.502708 19593 external_mini_cluster.cc:1428] Setting key 4a11a4f65bd47b845bc1b542d0c67c14
I20260619 14:52:13.502686 27115 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:55979
I20260619 14:52:13.509469 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.502389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43059 (local address 127.19.34.66:36565)
0619 14:52:13.502663 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:13.502678 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:13.502760 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:13.503181 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:13.503183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:13.503409 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:13.503516 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.503523 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.504348 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.504351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.504899 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.504907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.505339 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.505343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.505433 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.505880 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:13.505890 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:13.506185 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:13.507994 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:13.507996 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:13.507998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:13.508183 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:13.508554 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:13.508557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:13.508558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:13.508640 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:13.509111 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:13.509113 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:13.509185 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 14:52:13.509260 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 14:52:13.509351 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":100,"threads_started":1}
I20260619 14:52:13.509701 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.502713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:36565 (local address 127.19.34.126:43059)
0619 14:52:13.502818 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:13.502821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:13.502835 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:52:13.502869 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:13.502872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:13.502912 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:13.502988 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:13.503642 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.504197 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.505013 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.505230 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.505337 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.506309 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:13.506313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:13.506315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:13.506337 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:13.507887 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:13.508350 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:13.508357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:13.508360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:13.508454 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:13.508732 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:13.508738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:13.508742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:13.509011 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:13.509112 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:13.509347 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 14:52:13.509473 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for oryx/127.19.34.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.34.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.34.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.67.keytab.
Entry for principal oryx/127.19.34.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.67.keytab.
I20260619 14:52:13.528856 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:43059
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.67.keytab
--principal=oryx/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:40509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:13.633680 27469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:13.633891 27469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:13.633963 27469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:13.637910 27469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:13.637957 27469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:13.637992 27469 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:52:13.638022 27469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:52:13.642627 27469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:40509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.67.keytab
--principal=oryx/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.34.126:43059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:13.643466 27469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:13.644490 27469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:13.653723 27475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:13.653771 27469 server_base.cc:1034] running on GCE node
W20260619 14:52:13.653712 27474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:13.653725 27477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:13.654260 27469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:13.654805 27469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:13.655985 27469 hybrid_clock.cc:648] HybridClock initialized: now 1781880733655964 us; error 51 us; skew 500 ppm
Jun 19 14:52:13 dist-test-slave-3qdh krb5kdc[27072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880733, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:13.658547 27469 init.cc:377] Logged in from keytab as oryx/127.19.34.67@KRBTEST.COM (short username oryx)
I20260619 14:52:13.659288 27469 webserver.cc:460] Webserver started at http://127.19.34.67:41431/ using document root <none> and password file <none>
I20260619 14:52:13.659734 27469 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:13.659787 27469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:13.659930 27469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:13.661741 27469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "b9f8988d50b4448ea65bc250030db5bd"
format_stamp: "Formatted at 2026-06-19 14:52:13 on dist-test-slave-3qdh"
server_key: "ebe0012225e7eba79eac6e02f4b3dca6"
server_key_iv: "cff0307868e482f2518e4c2130f38cd8"
server_key_version: "encryptionkey@0"
I20260619 14:52:13.662204 27469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "b9f8988d50b4448ea65bc250030db5bd"
format_stamp: "Formatted at 2026-06-19 14:52:13 on dist-test-slave-3qdh"
server_key: "ebe0012225e7eba79eac6e02f4b3dca6"
server_key_iv: "cff0307868e482f2518e4c2130f38cd8"
server_key_version: "encryptionkey@0"
I20260619 14:52:13.665721 27469 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 14:52:13.667799 27484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:13.668987 27469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:13.669137 27469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "b9f8988d50b4448ea65bc250030db5bd"
format_stamp: "Formatted at 2026-06-19 14:52:13 on dist-test-slave-3qdh"
server_key: "ebe0012225e7eba79eac6e02f4b3dca6"
server_key_iv: "cff0307868e482f2518e4c2130f38cd8"
server_key_version: "encryptionkey@0"
I20260619 14:52:13.669230 27469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:13.680413 27469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:13.682883 27469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:13.683048 27469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:13.684403 27469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:13.684448 27469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:13.684502 27469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:13.684535 27469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:52:13 dist-test-slave-3qdh krb5kdc[27072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880733, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM
I20260619 14:52:13.694985 27469 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:41781
I20260619 14:52:13.695091 27599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:41781 every 8 connection(s)
I20260619 14:52:13.695914 27469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:52:13.697642 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.685102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:44437 (local address 127.19.34.126:43059)
0619 14:52:13.685270 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:13.685275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:13.686028 (+ 753us) server_negotiation.cc:407] Connection header received
0619 14:52:13.686918 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:13.686921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:13.686990 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:13.687113 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:13.688803 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.689439 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.690509 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.690739 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.690846 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.694057 (+ 3211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:13.694060 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:13.694062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:13.694082 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:13.695648 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:13.696187 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:13.696190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:13.696191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:13.696291 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:13.696724 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:13.696727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:13.696729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:13.696921 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:13.696995 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:13.697427 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 14:52:13.697517 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:52:13.698343 27493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.685468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43059 (local address 127.19.34.67:44437)
0619 14:52:13.685874 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:13.685919 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:13.686728 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:13.687419 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:13.687428 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:13.687951 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:13.688636 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.688658 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.689559 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.689564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.690383 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.690392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.690888 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.690893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.691061 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.691637 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:13.691663 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:13.693906 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:13.695758 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:13.695763 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:13.695775 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:13.696095 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:13.696404 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:13.696409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:13.696411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:13.696572 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:13.697012 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:13.697016 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:13.697216 (+ 200us) client_negotiation.cc:769] Sending connection context
0619 14:52:13.697362 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 14:52:13.697588 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":120,"threads_started":1}
I20260619 14:52:13.699378 27600 heartbeater.cc:347] Connected to a master server at 127.19.34.126:43059
I20260619 14:52:13.699560 27600 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:13.700011 27600 heartbeater.cc:511] Master 127.19.34.126:43059 requested a full tablet report, sending...
I20260619 14:52:13.701105 27117 ts_manager.cc:194] Registered new tserver with Master: b9f8988d50b4448ea65bc250030db5bd (127.19.34.67:41781)
I20260619 14:52:13.701683 27117 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:44437
I20260619 14:52:13.704087 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 27469
I20260619 14:52:13.704200 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:52:13.704473 19593 external_mini_cluster.cc:1428] Setting key c1ca2b080fcdc18db4864428de99f68c
I20260619 14:52:13.706256 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:52:13.708375 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.701701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:34075 (local address 127.19.34.126:43059)
0619 14:52:13.701885 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:13.701892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:13.701935 (+ 43us) server_negotiation.cc:407] Connection header received
0619 14:52:13.702094 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:13.702097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:13.702134 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:13.702218 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:13.702621 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.703187 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.704043 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.704348 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.704441 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.705456 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:13.705459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:13.705462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:13.705483 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:13.706858 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:13.707241 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:13.707245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:13.707247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:13.707317 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:13.707590 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:13.707595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:13.707596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:13.707754 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:13.707866 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:13.708109 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 14:52:13.708203 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260619 14:52:13.708384 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.701603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43059 (local address 127.19.34.67:34075)
0619 14:52:13.701857 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:13.701870 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:13.701990 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:13.702232 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:13.702235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:13.702409 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:13.702492 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.702497 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.703302 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.703305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.703935 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.703944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.704452 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.704455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.704546 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.705011 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:13.705030 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:13.705306 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:13.706964 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:13.706970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:13.706972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:13.707151 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:13.707401 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:13.707405 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:13.707407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:13.707504 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:13.707858 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:13.707861 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:13.707945 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 14:52:13.708050 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 14:52:13.708181 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":104,"threads_started":1}
Jun 19 14:52:13 dist-test-slave-3qdh krb5kdc[27072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880733, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:52:13 dist-test-slave-3qdh krb5kdc[27072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:13 dist-test-slave-3qdh krb5kdc[27072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:52:13.727679 27166 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:13.722254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35938 (local address 127.19.34.126:43059)
0619 14:52:13.722364 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:13.722367 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:13.722378 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:52:13.722418 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:13.722420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:13.722449 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:13.722510 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:13.723120 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.723534 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.724259 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.724503 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.724576 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.727560 (+ 2984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35938: BlockingRecv error: recv got EOF from 127.0.0.1:35938 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 14:52:13 dist-test-slave-3qdh krb5kdc[27072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880733, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM
I20260619 14:52:13.736259 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.728836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35942 (local address 127.19.34.126:43059)
0619 14:52:13.728931 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:13.728934 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:13.729013 (+ 79us) server_negotiation.cc:407] Connection header received
0619 14:52:13.729096 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:13.729098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:13.729126 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:13.729188 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:13.729681 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.730111 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.730829 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.731018 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.731099 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.733259 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:13.733262 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:13.733264 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:13.733283 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:13.734506 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:13.734974 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:13.734980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:13.734982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:13.735089 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:13.735386 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:13.735390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:13.735392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:13.735594 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:13.735737 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:13.735982 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 14:52:13.736107 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:52:13.738756 27117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:52:13.741636 27117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:52:13.755546 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.749862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:43377 (local address 127.0.0.1:33426)
0619 14:52:13.750394 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:13.750407 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:13.750479 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:13.751467 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:13.751470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:13.751496 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:13.751609 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.751617 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.752724 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.752728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.754123 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.754133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.754962 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.754966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.755086 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.755132 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 14:52:13.755280 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 14:52:13.755403 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":136,"thread_start_us":104,"threads_started":1}
I20260619 14:52:13.755558 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.750473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33426 (local address 127.19.34.66:43377)
0619 14:52:13.751004 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:13.751019 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:13.751045 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:52:13.751126 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:13.751133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:13.751317 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:13.751473 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:13.751735 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.752612 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.754254 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.754832 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.754964 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.755136 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:13.755310 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 14:52:13.755403 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":106,"threads_started":1}
I20260619 14:52:13.755761 27622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.749654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:41781 (local address 127.0.0.1:59850)
0619 14:52:13.749970 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:13.750265 (+ 295us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:13.750382 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:13.751309 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:13.751314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:13.751344 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:13.751429 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.751434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.752760 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.752763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.754157 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.754167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.755375 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.755379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.755467 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.755489 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:52:13.755559 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 14:52:13.755625 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":70,"thread_start_us":62,"threads_started":1}
I20260619 14:52:13.756156 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.749802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59850 (local address 127.19.34.67:41781)
0619 14:52:13.750668 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:13.750683 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:13.750706 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:52:13.750784 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:13.750790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:13.750989 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:13.751173 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:13.751543 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.752634 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.754285 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.755069 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.755177 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.755315 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:13.755918 (+ 603us) server_negotiation.cc:299] Negotiation successful
0619 14:52:13.756005 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":117,"threads_started":1}
I20260619 14:52:13.756541 27626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.750525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34119 (local address 127.0.0.1:44702)
0619 14:52:13.751669 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:13.751681 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:13.751759 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:13.752225 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:13.752228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:13.752247 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:13.752367 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.752374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.753691 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.753694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.754846 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.754856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.756079 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.756082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.756179 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.756210 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 14:52:13.756320 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 14:52:13.756383 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"mutex_wait_us":24,"thread_start_us":120,"threads_started":1}
I20260619 14:52:13.756631 27627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.750815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44702 (local address 127.19.34.65:34119)
0619 14:52:13.751205 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:13.751216 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:13.751850 (+ 634us) server_negotiation.cc:407] Connection header received
0619 14:52:13.751926 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:13.751930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:13.752082 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:13.752220 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:13.752480 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.753549 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.754984 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.755777 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.755882 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.756020 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:13.756380 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 14:52:13.756473 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":103,"threads_started":1}
I20260619 14:52:13.759183 27394 tablet_service.cc:1467] Processing CreateTablet for tablet 87924b9bc64c49d2b21e8e59339eea8e (DEFAULT_TABLE table=test-table [id=c66d5d229734414fb12c21ecd02090fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:13.759225 27254 tablet_service.cc:1467] Processing CreateTablet for tablet 87924b9bc64c49d2b21e8e59339eea8e (DEFAULT_TABLE table=test-table [id=c66d5d229734414fb12c21ecd02090fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:13.759227 27534 tablet_service.cc:1467] Processing CreateTablet for tablet 87924b9bc64c49d2b21e8e59339eea8e (DEFAULT_TABLE table=test-table [id=c66d5d229734414fb12c21ecd02090fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:13.760097 27394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87924b9bc64c49d2b21e8e59339eea8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:13.760154 27254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87924b9bc64c49d2b21e8e59339eea8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:13.760154 27534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87924b9bc64c49d2b21e8e59339eea8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:13.766749 27628 tablet_bootstrap.cc:492] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871: Bootstrap starting.
I20260619 14:52:13.768488 27629 tablet_bootstrap.cc:492] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92: Bootstrap starting.
I20260619 14:52:13.768631 27630 tablet_bootstrap.cc:492] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd: Bootstrap starting.
I20260619 14:52:13.769129 27628 tablet_bootstrap.cc:654] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:13.770000 27628 log.cc:824] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:13.770429 27630 tablet_bootstrap.cc:654] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:13.770761 27629 tablet_bootstrap.cc:654] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:13.771206 27630 log.cc:824] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:13.771555 27629 log.cc:824] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:13.772145 27628 tablet_bootstrap.cc:492] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871: No bootstrap required, opened a new log
I20260619 14:52:13.772377 27628 ts_tablet_manager.cc:1397] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:52:13.773214 27630 tablet_bootstrap.cc:492] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd: No bootstrap required, opened a new log
I20260619 14:52:13.773439 27630 ts_tablet_manager.cc:1397] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:52:13.774039 27629 tablet_bootstrap.cc:492] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92: No bootstrap required, opened a new log
I20260619 14:52:13.774214 27629 ts_tablet_manager.cc:1397] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:52:13.776409 27630 raft_consensus.cc:348] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } }
I20260619 14:52:13.776459 27628 raft_consensus.cc:348] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } }
I20260619 14:52:13.776641 27630 raft_consensus.cc:374] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:13.776692 27630 raft_consensus.cc:729] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f8988d50b4448ea65bc250030db5bd, State: Initialized, Role: FOLLOWER
I20260619 14:52:13.776705 27628 raft_consensus.cc:374] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:13.776754 27628 raft_consensus.cc:729] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01932012b1bd4b8cb18f21da85bf1871, State: Initialized, Role: FOLLOWER
I20260619 14:52:13.777112 27630 consensus_queue.cc:260] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } }
I20260619 14:52:13.777186 27628 consensus_queue.cc:260] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } }
I20260619 14:52:13.778208 27630 ts_tablet_manager.cc:1428] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:52:13.778509 27629 raft_consensus.cc:348] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } }
I20260619 14:52:13.778803 27629 raft_consensus.cc:374] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:13.778867 27629 raft_consensus.cc:729] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf2f87cb790496882ab5c0299489e92, State: Initialized, Role: FOLLOWER
I20260619 14:52:13.778954 27600 heartbeater.cc:503] Master 127.19.34.126:43059 was elected leader, sending a full tablet report...
I20260619 14:52:13.779269 27629 consensus_queue.cc:260] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } }
I20260619 14:52:13.779892 27320 heartbeater.cc:503] Master 127.19.34.126:43059 was elected leader, sending a full tablet report...
I20260619 14:52:13.780246 27629 ts_tablet_manager.cc:1428] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:52:13.780629 27628 ts_tablet_manager.cc:1428] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260619 14:52:13.780921 27460 heartbeater.cc:503] Master 127.19.34.126:43059 was elected leader, sending a full tablet report...
W20260619 14:52:13.800989 27321 tablet.cc:2369] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:52:13.947008 27601 tablet.cc:2369] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:52:13.986033 27636 raft_consensus.cc:482] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:52:13.986303 27636 raft_consensus.cc:504] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } }
I20260619 14:52:13.987545 27636 leader_election.cc:288] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9f8988d50b4448ea65bc250030db5bd (127.19.34.67:41781), 01932012b1bd4b8cb18f21da85bf1871 (127.19.34.66:43377)
I20260619 14:52:13.991509 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.988033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:43787 (local address 127.19.34.66:43377)
0619 14:52:13.988146 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:13.988159 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:13.988188 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:52:13.988391 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:13.988394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:13.988457 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:13.988528 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:13.988835 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.989389 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.990520 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.990903 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.991022 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.991098 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:13.991295 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 14:52:13.991387 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:52:13.991580 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.987847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:50921 (local address 127.19.34.67:41781)
0619 14:52:13.987991 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:13.987997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:13.988068 (+ 71us) server_negotiation.cc:407] Connection header received
0619 14:52:13.988279 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:13.988284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:13.988377 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:13.988453 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:13.988781 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.989397 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.990520 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.990937 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.991125 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.991182 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:13.991389 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 14:52:13.991461 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:52:13.991550 27637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.987695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:41781 (local address 127.19.34.65:50921)
0619 14:52:13.987984 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:13.987999 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:13.988145 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:13.988483 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:13.988489 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:13.988513 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:13.988665 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.988671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.989522 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.989526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.990385 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.990393 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.991125 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.991129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.991219 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.991262 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 14:52:13.991340 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 14:52:13.991413 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":66,"thread_start_us":87,"threads_started":1}
I20260619 14:52:13.991554 27638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:13.987844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:43377 (local address 127.19.34.65:43787)
0619 14:52:13.988123 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:13.988136 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:13.988233 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:13.988570 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:13.988574 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:13.988597 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:13.988694 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.988699 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.989509 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.989516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.990384 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:13.990393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:13.991050 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:13.991057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:13.991159 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:13.991182 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:52:13.991267 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 14:52:13.991371 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
I20260619 14:52:13.992518 27554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87924b9bc64c49d2b21e8e59339eea8e" candidate_uuid: "1cf2f87cb790496882ab5c0299489e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9f8988d50b4448ea65bc250030db5bd" is_pre_election: true
I20260619 14:52:13.992551 27412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87924b9bc64c49d2b21e8e59339eea8e" candidate_uuid: "1cf2f87cb790496882ab5c0299489e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01932012b1bd4b8cb18f21da85bf1871" is_pre_election: true
I20260619 14:52:13.992820 27412 raft_consensus.cc:2454] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cf2f87cb790496882ab5c0299489e92 in term 0.
I20260619 14:52:13.992820 27554 raft_consensus.cc:2454] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cf2f87cb790496882ab5c0299489e92 in term 0.
I20260619 14:52:13.993270 27205 leader_election.cc:302] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cf2f87cb790496882ab5c0299489e92, b9f8988d50b4448ea65bc250030db5bd; no voters:
I20260619 14:52:13.993554 27636 raft_consensus.cc:2781] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:52:13.993649 27636 raft_consensus.cc:482] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:52:13.993696 27636 raft_consensus.cc:3037] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:13.994722 27636 raft_consensus.cc:504] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } }
I20260619 14:52:13.995059 27636 leader_election.cc:288] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 [CANDIDATE]: Term 1 election: Requested vote from peers b9f8988d50b4448ea65bc250030db5bd (127.19.34.67:41781), 01932012b1bd4b8cb18f21da85bf1871 (127.19.34.66:43377)
I20260619 14:52:13.995393 27554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87924b9bc64c49d2b21e8e59339eea8e" candidate_uuid: "1cf2f87cb790496882ab5c0299489e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9f8988d50b4448ea65bc250030db5bd"
I20260619 14:52:13.995442 27412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87924b9bc64c49d2b21e8e59339eea8e" candidate_uuid: "1cf2f87cb790496882ab5c0299489e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01932012b1bd4b8cb18f21da85bf1871"
I20260619 14:52:13.995529 27554 raft_consensus.cc:3037] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:13.995570 27412 raft_consensus.cc:3037] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:13.996698 27412 raft_consensus.cc:2454] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf2f87cb790496882ab5c0299489e92 in term 1.
I20260619 14:52:13.997007 27207 leader_election.cc:302] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01932012b1bd4b8cb18f21da85bf1871, 1cf2f87cb790496882ab5c0299489e92; no voters:
I20260619 14:52:13.997162 27636 raft_consensus.cc:2781] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:13.997294 27554 raft_consensus.cc:2454] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf2f87cb790496882ab5c0299489e92 in term 1.
I20260619 14:52:13.997331 27636 raft_consensus.cc:686] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 [term 1 LEADER]: Becoming Leader. State: Replica: 1cf2f87cb790496882ab5c0299489e92, State: Running, Role: LEADER
I20260619 14:52:13.997664 27636 consensus_queue.cc:237] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } }
W20260619 14:52:13.998019 27461 tablet.cc:2369] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:52:14.000922 27117 catalog_manager.cc:5495] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cf2f87cb790496882ab5c0299489e92 (127.19.34.65). New cstate: current_term: 1 leader_uuid: "1cf2f87cb790496882ab5c0299489e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } health_report { overall_health: HEALTHY } } }
I20260619 14:52:14.050673 27627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.047197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44718 (local address 127.19.34.65:34119)
0619 14:52:14.047323 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:14.047327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:14.047341 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:52:14.047417 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:14.047420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:14.047477 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:14.047576 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:52:14.047857 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.048367 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.049229 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.049537 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.049647 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.049919 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:52:14.050307 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:52:14.050419 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:14.050503 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 14:52:14.050550 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:52:14.055833 27412 raft_consensus.cc:1261] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871 [term 1 FOLLOWER]: Refusing update from remote peer 1cf2f87cb790496882ab5c0299489e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:14.055845 27554 raft_consensus.cc:1261] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd [term 1 FOLLOWER]: Refusing update from remote peer 1cf2f87cb790496882ab5c0299489e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:14.056499 27636 consensus_queue.cc:1035] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:14.056739 27639 consensus_queue.cc:1035] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:14.062956 27642 mvcc.cc:205] Tried to move back new op lower bound from 7298583486686924800 to 7298583486457315328. Current Snapshot: MvccSnapshot[applied={T|T < 7298583486686924800}]
I20260619 14:52:14.063316 27643 mvcc.cc:205] Tried to move back new op lower bound from 7298583486686924800 to 7298583486457315328. Current Snapshot: MvccSnapshot[applied={T|T < 7298583486686924800}]
I20260619 14:52:14.072007 27115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:14.072190 27115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:14.074328 27115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16: Sending DeleteTablet for 3 replicas of tablet 87924b9bc64c49d2b21e8e59339eea8e
I20260619 14:52:14.074939 27534 tablet_service.cc:1514] Processing DeleteTablet for tablet 87924b9bc64c49d2b21e8e59339eea8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:14 UTC) from {username='oryx', principal='oryx/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:59850
I20260619 14:52:14.074983 27394 tablet_service.cc:1514] Processing DeleteTablet for tablet 87924b9bc64c49d2b21e8e59339eea8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:14 UTC) from {username='oryx', principal='oryx/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:33426
I20260619 14:52:14.075336 27652 tablet_replica.cc:330] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871: stopping tablet replica
I20260619 14:52:14.075625 27653 tablet_replica.cc:330] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd: stopping tablet replica
I20260619 14:52:14.075667 27652 raft_consensus.cc:2229] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:14.075896 27653 raft_consensus.cc:2229] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:14.076009 27254 tablet_service.cc:1514] Processing DeleteTablet for tablet 87924b9bc64c49d2b21e8e59339eea8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:14 UTC) from {username='oryx', principal='oryx/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:44702
I20260619 14:52:14.076066 27652 raft_consensus.cc:2258] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:14.076166 27653 raft_consensus.cc:2258] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:14.076586 27115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 14:52:14.076761 27654 tablet_replica.cc:330] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92: stopping tablet replica
W20260619 14:52:14.076931 27115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:52:14.076958 27654 raft_consensus.cc:2229] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:52:14.077256 27654 raft_consensus.cc:2258] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:14.077791 27652 ts_tablet_manager.cc:1905] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:14.078442 27653 ts_tablet_manager.cc:1905] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:14.080500 27654 ts_tablet_manager.cc:1905] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:14.081233 27254 tablet_service.cc:1467] Processing CreateTablet for tablet b78fb165207b43238a3c0b616456c7d7 (DEFAULT_TABLE table=test-table [id=0c09855a47ee4d19883c2e747f3f69d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:14.081446 27254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b78fb165207b43238a3c0b616456c7d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:14.081506 27394 tablet_service.cc:1467] Processing CreateTablet for tablet b78fb165207b43238a3c0b616456c7d7 (DEFAULT_TABLE table=test-table [id=0c09855a47ee4d19883c2e747f3f69d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:14.081746 27394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b78fb165207b43238a3c0b616456c7d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:14.083525 27629 tablet_bootstrap.cc:492] T b78fb165207b43238a3c0b616456c7d7 P 1cf2f87cb790496882ab5c0299489e92: Bootstrap starting.
I20260619 14:52:14.083740 27628 tablet_bootstrap.cc:492] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871: Bootstrap starting.
I20260619 14:52:14.083925 27534 tablet_service.cc:1467] Processing CreateTablet for tablet b78fb165207b43238a3c0b616456c7d7 (DEFAULT_TABLE table=test-table [id=0c09855a47ee4d19883c2e747f3f69d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:14.084093 27654 ts_tablet_manager.cc:1918] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:14.084156 27534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b78fb165207b43238a3c0b616456c7d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:14.084187 27654 log.cc:1196] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-0/wal/wals/87924b9bc64c49d2b21e8e59339eea8e
I20260619 14:52:14.084493 27629 tablet_bootstrap.cc:654] T b78fb165207b43238a3c0b616456c7d7 P 1cf2f87cb790496882ab5c0299489e92: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:14.084522 27654 ts_tablet_manager.cc:1939] T 87924b9bc64c49d2b21e8e59339eea8e P 1cf2f87cb790496882ab5c0299489e92: Deleting consensus metadata
I20260619 14:52:14.084609 27628 tablet_bootstrap.cc:654] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:14.085567 27104 catalog_manager.cc:4841] TS 1cf2f87cb790496882ab5c0299489e92 (127.19.34.65:34119): tablet 87924b9bc64c49d2b21e8e59339eea8e (table test-table [id=c66d5d229734414fb12c21ecd02090fe]) successfully deleted
I20260619 14:52:14.085700 27653 ts_tablet_manager.cc:1918] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:14.085798 27653 log.cc:1196] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/87924b9bc64c49d2b21e8e59339eea8e
I20260619 14:52:14.086098 27653 ts_tablet_manager.cc:1939] T 87924b9bc64c49d2b21e8e59339eea8e P b9f8988d50b4448ea65bc250030db5bd: Deleting consensus metadata
I20260619 14:52:14.086390 27630 tablet_bootstrap.cc:492] T b78fb165207b43238a3c0b616456c7d7 P b9f8988d50b4448ea65bc250030db5bd: Bootstrap starting.
I20260619 14:52:14.087152 27630 tablet_bootstrap.cc:654] T b78fb165207b43238a3c0b616456c7d7 P b9f8988d50b4448ea65bc250030db5bd: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:14.088212 27630 tablet_bootstrap.cc:492] T b78fb165207b43238a3c0b616456c7d7 P b9f8988d50b4448ea65bc250030db5bd: No bootstrap required, opened a new log
I20260619 14:52:14.088281 27630 ts_tablet_manager.cc:1397] T b78fb165207b43238a3c0b616456c7d7 P b9f8988d50b4448ea65bc250030db5bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:14.088330 27652 ts_tablet_manager.cc:1918] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:14.088399 27652 log.cc:1196] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-1/wal/wals/87924b9bc64c49d2b21e8e59339eea8e
I20260619 14:52:14.088608 27101 catalog_manager.cc:4841] TS b9f8988d50b4448ea65bc250030db5bd (127.19.34.67:41781): tablet 87924b9bc64c49d2b21e8e59339eea8e (table test-table [id=c66d5d229734414fb12c21ecd02090fe]) successfully deleted
I20260619 14:52:14.088714 27630 raft_consensus.cc:348] T b78fb165207b43238a3c0b616456c7d7 P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.088824 27630 raft_consensus.cc:374] T b78fb165207b43238a3c0b616456c7d7 P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:14.088853 27630 raft_consensus.cc:729] T b78fb165207b43238a3c0b616456c7d7 P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f8988d50b4448ea65bc250030db5bd, State: Initialized, Role: FOLLOWER
I20260619 14:52:14.088927 27652 ts_tablet_manager.cc:1939] T 87924b9bc64c49d2b21e8e59339eea8e P 01932012b1bd4b8cb18f21da85bf1871: Deleting consensus metadata
I20260619 14:52:14.088963 27630 consensus_queue.cc:260] T b78fb165207b43238a3c0b616456c7d7 P b9f8988d50b4448ea65bc250030db5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.089080 27629 tablet_bootstrap.cc:492] T b78fb165207b43238a3c0b616456c7d7 P 1cf2f87cb790496882ab5c0299489e92: No bootstrap required, opened a new log
I20260619 14:52:14.089152 27629 ts_tablet_manager.cc:1397] T b78fb165207b43238a3c0b616456c7d7 P 1cf2f87cb790496882ab5c0299489e92: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260619 14:52:14.089219 27630 ts_tablet_manager.cc:1428] T b78fb165207b43238a3c0b616456c7d7 P b9f8988d50b4448ea65bc250030db5bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:52:14.089525 27629 raft_consensus.cc:348] T b78fb165207b43238a3c0b616456c7d7 P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.089632 27629 raft_consensus.cc:374] T b78fb165207b43238a3c0b616456c7d7 P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:14.089669 27629 raft_consensus.cc:729] T b78fb165207b43238a3c0b616456c7d7 P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf2f87cb790496882ab5c0299489e92, State: Initialized, Role: FOLLOWER
I20260619 14:52:14.089768 27629 consensus_queue.cc:260] T b78fb165207b43238a3c0b616456c7d7 P 1cf2f87cb790496882ab5c0299489e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.090030 27629 ts_tablet_manager.cc:1428] T b78fb165207b43238a3c0b616456c7d7 P 1cf2f87cb790496882ab5c0299489e92: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 14:52:14.090178 27103 catalog_manager.cc:4841] TS 01932012b1bd4b8cb18f21da85bf1871 (127.19.34.66:43377): tablet 87924b9bc64c49d2b21e8e59339eea8e (table test-table [id=c66d5d229734414fb12c21ecd02090fe]) successfully deleted
I20260619 14:52:14.096907 27628 tablet_bootstrap.cc:492] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871: No bootstrap required, opened a new log
I20260619 14:52:14.096972 27628 ts_tablet_manager.cc:1397] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260619 14:52:14.097306 27628 raft_consensus.cc:348] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.097384 27628 raft_consensus.cc:374] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:14.097404 27628 raft_consensus.cc:729] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01932012b1bd4b8cb18f21da85bf1871, State: Initialized, Role: FOLLOWER
I20260619 14:52:14.097550 27628 consensus_queue.cc:260] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.097683 27635 raft_consensus.cc:482] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:52:14.097765 27635 raft_consensus.cc:504] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.097891 27628 ts_tablet_manager.cc:1428] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:52:14.098701 27635 leader_election.cc:288] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cf2f87cb790496882ab5c0299489e92 (127.19.34.65:34119), b9f8988d50b4448ea65bc250030db5bd (127.19.34.67:41781)
I20260619 14:52:14.102571 27660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.099212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:41781 (local address 127.19.34.66:39697)
0619 14:52:14.099411 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:14.099425 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:14.099517 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:14.099777 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:14.099780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:14.099793 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:14.099903 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:14.099911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.100829 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.100831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:14.101548 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:14.101555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.102179 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.102183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:14.102282 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.102298 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 14:52:14.102378 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 14:52:14.102437 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
I20260619 14:52:14.102548 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.099246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:39697 (local address 127.19.34.67:41781)
0619 14:52:14.099594 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:14.099598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:14.099617 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:52:14.099657 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:14.099659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:14.099687 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:14.099768 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:14.100022 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.100686 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.101676 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.102029 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.102176 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.102225 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:14.102392 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 14:52:14.102450 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278}
I20260619 14:52:14.103001 27554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b78fb165207b43238a3c0b616456c7d7" candidate_uuid: "01932012b1bd4b8cb18f21da85bf1871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9f8988d50b4448ea65bc250030db5bd" is_pre_election: true
I20260619 14:52:14.103137 27554 raft_consensus.cc:2454] T b78fb165207b43238a3c0b616456c7d7 P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01932012b1bd4b8cb18f21da85bf1871 in term 0.
I20260619 14:52:14.103363 27659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.098970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34119 (local address 127.19.34.66:42767)
0619 14:52:14.099272 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:14.099285 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:14.099365 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:14.099836 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:14.099839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:14.099853 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:14.099987 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:14.099992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.100946 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.100949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:14.102029 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:14.102038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.102964 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.102968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:14.103066 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.103085 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:52:14.103179 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 14:52:14.103240 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":107,"threads_started":1}
I20260619 14:52:14.103392 27627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.099462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:42767 (local address 127.19.34.65:34119)
0619 14:52:14.099557 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:14.099561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:14.099572 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:52:14.099607 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:14.099610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:14.099645 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:14.099739 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:14.100092 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.100796 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.102183 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.102745 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.102823 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.102865 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:14.103204 (+ 339us) server_negotiation.cc:299] Negotiation successful
0619 14:52:14.103281 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 14:52:14.103461 27345 leader_election.cc:302] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01932012b1bd4b8cb18f21da85bf1871, b9f8988d50b4448ea65bc250030db5bd; no voters:
I20260619 14:52:14.103711 27635 raft_consensus.cc:2781] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:52:14.103760 27635 raft_consensus.cc:482] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:52:14.103798 27635 raft_consensus.cc:3037] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:14.104487 27274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b78fb165207b43238a3c0b616456c7d7" candidate_uuid: "01932012b1bd4b8cb18f21da85bf1871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cf2f87cb790496882ab5c0299489e92" is_pre_election: true
I20260619 14:52:14.104673 27274 raft_consensus.cc:2454] T b78fb165207b43238a3c0b616456c7d7 P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01932012b1bd4b8cb18f21da85bf1871 in term 0.
I20260619 14:52:14.104727 27635 raft_consensus.cc:504] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.105005 27635 leader_election.cc:288] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 [CANDIDATE]: Term 1 election: Requested vote from peers 1cf2f87cb790496882ab5c0299489e92 (127.19.34.65:34119), b9f8988d50b4448ea65bc250030db5bd (127.19.34.67:41781)
I20260619 14:52:14.105314 27554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b78fb165207b43238a3c0b616456c7d7" candidate_uuid: "01932012b1bd4b8cb18f21da85bf1871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9f8988d50b4448ea65bc250030db5bd"
I20260619 14:52:14.105468 27554 raft_consensus.cc:3037] T b78fb165207b43238a3c0b616456c7d7 P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:14.105466 27274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b78fb165207b43238a3c0b616456c7d7" candidate_uuid: "01932012b1bd4b8cb18f21da85bf1871" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cf2f87cb790496882ab5c0299489e92"
I20260619 14:52:14.105556 27274 raft_consensus.cc:3037] T b78fb165207b43238a3c0b616456c7d7 P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:14.106420 27554 raft_consensus.cc:2454] T b78fb165207b43238a3c0b616456c7d7 P b9f8988d50b4448ea65bc250030db5bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01932012b1bd4b8cb18f21da85bf1871 in term 1.
I20260619 14:52:14.106597 27274 raft_consensus.cc:2454] T b78fb165207b43238a3c0b616456c7d7 P 1cf2f87cb790496882ab5c0299489e92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01932012b1bd4b8cb18f21da85bf1871 in term 1.
I20260619 14:52:14.106746 27345 leader_election.cc:302] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01932012b1bd4b8cb18f21da85bf1871, b9f8988d50b4448ea65bc250030db5bd; no voters:
I20260619 14:52:14.106899 27635 raft_consensus.cc:2781] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:14.107019 27635 raft_consensus.cc:686] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 [term 1 LEADER]: Becoming Leader. State: Replica: 01932012b1bd4b8cb18f21da85bf1871, State: Running, Role: LEADER
I20260619 14:52:14.107247 27635 consensus_queue.cc:237] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.109251 27117 catalog_manager.cc:5495] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01932012b1bd4b8cb18f21da85bf1871 (127.19.34.66). New cstate: current_term: 1 leader_uuid: "01932012b1bd4b8cb18f21da85bf1871" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } health_report { overall_health: UNKNOWN } } }
Jun 19 14:52:14 dist-test-slave-3qdh krb5kdc[27072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880732, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.126@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM
I20260619 14:52:14.141671 27667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.133579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43059 (local address 127.0.0.1:35958)
0619 14:52:14.133857 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:14.133868 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:14.133939 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:14.134163 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:14.134165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:14.134462 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:14.134535 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:14.134540 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.135472 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.135474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:14.135998 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:14.136005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.136613 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.136616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:14.136694 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.137252 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:14.137262 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:14.138483 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:14.140269 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:14.140276 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:14.140280 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:14.140540 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:14.140815 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:14.140816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:14.140817 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:14.140921 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:14.141289 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:14.141293 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:14.141376 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 14:52:14.141462 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 14:52:14.141555 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":113,"threads_started":1}
I20260619 14:52:14.141860 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.133680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35958 (local address 127.19.34.126:43059)
0619 14:52:14.133794 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:14.133798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:14.133929 (+ 131us) server_negotiation.cc:407] Connection header received
0619 14:52:14.134008 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:14.134011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:14.134048 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:14.134119 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:14.134635 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.135357 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.136108 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.136463 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.136561 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.138600 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:14.138604 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:14.138606 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:14.138628 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:14.140149 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:14.140642 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:14.140645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:14.140647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:14.140733 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:14.141000 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:14.141003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:14.141005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:14.141191 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:14.141282 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:14.141590 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 14:52:14.141709 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:52:14.145318 27117 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:35958:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 14:52:14.145969 27117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:52:14.150444 27254 tablet_service.cc:1467] Processing CreateTablet for tablet d4bcec8881bc4ce5a45cba85ca766552 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=74a0c89373ce4befa130e48afacecdd4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:52:14.150763 27254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4bcec8881bc4ce5a45cba85ca766552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:14.150679 27394 tablet_service.cc:1467] Processing CreateTablet for tablet d4bcec8881bc4ce5a45cba85ca766552 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=74a0c89373ce4befa130e48afacecdd4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:52:14.150983 27394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4bcec8881bc4ce5a45cba85ca766552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:14.152798 27629 tablet_bootstrap.cc:492] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92: Bootstrap starting.
I20260619 14:52:14.153066 27534 tablet_service.cc:1467] Processing CreateTablet for tablet d4bcec8881bc4ce5a45cba85ca766552 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=74a0c89373ce4befa130e48afacecdd4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:52:14.153381 27534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4bcec8881bc4ce5a45cba85ca766552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:14.153558 27628 tablet_bootstrap.cc:492] T d4bcec8881bc4ce5a45cba85ca766552 P 01932012b1bd4b8cb18f21da85bf1871: Bootstrap starting.
I20260619 14:52:14.153647 27629 tablet_bootstrap.cc:654] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:14.154461 27628 tablet_bootstrap.cc:654] T d4bcec8881bc4ce5a45cba85ca766552 P 01932012b1bd4b8cb18f21da85bf1871: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:14.154757 27629 tablet_bootstrap.cc:492] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92: No bootstrap required, opened a new log
I20260619 14:52:14.154827 27629 ts_tablet_manager.cc:1397] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:52:14.155181 27629 raft_consensus.cc:348] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.155278 27629 raft_consensus.cc:374] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:14.155306 27629 raft_consensus.cc:729] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf2f87cb790496882ab5c0299489e92, State: Initialized, Role: FOLLOWER
I20260619 14:52:14.155416 27629 consensus_queue.cc:260] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.155704 27629 ts_tablet_manager.cc:1428] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:52:14.155985 27628 tablet_bootstrap.cc:492] T d4bcec8881bc4ce5a45cba85ca766552 P 01932012b1bd4b8cb18f21da85bf1871: No bootstrap required, opened a new log
I20260619 14:52:14.156073 27628 ts_tablet_manager.cc:1397] T d4bcec8881bc4ce5a45cba85ca766552 P 01932012b1bd4b8cb18f21da85bf1871: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 14:52:14.156049 27649 tablet_replica.cc:439] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } } }
I20260619 14:52:14.156203 27649 tablet_replica.cc:442] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:52:14.156530 27628 raft_consensus.cc:348] T d4bcec8881bc4ce5a45cba85ca766552 P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.156654 27628 raft_consensus.cc:374] T d4bcec8881bc4ce5a45cba85ca766552 P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:14.156699 27628 raft_consensus.cc:729] T d4bcec8881bc4ce5a45cba85ca766552 P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01932012b1bd4b8cb18f21da85bf1871, State: Initialized, Role: FOLLOWER
I20260619 14:52:14.156819 27628 consensus_queue.cc:260] T d4bcec8881bc4ce5a45cba85ca766552 P 01932012b1bd4b8cb18f21da85bf1871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.157050 27628 ts_tablet_manager.cc:1428] T d4bcec8881bc4ce5a45cba85ca766552 P 01932012b1bd4b8cb18f21da85bf1871: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:52:14.157086 27658 tablet_replica.cc:439] T d4bcec8881bc4ce5a45cba85ca766552 P 01932012b1bd4b8cb18f21da85bf1871: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } } }
I20260619 14:52:14.157207 27658 tablet_replica.cc:442] T d4bcec8881bc4ce5a45cba85ca766552 P 01932012b1bd4b8cb18f21da85bf1871: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:52:14.157869 27630 tablet_bootstrap.cc:492] T d4bcec8881bc4ce5a45cba85ca766552 P b9f8988d50b4448ea65bc250030db5bd: Bootstrap starting.
I20260619 14:52:14.158619 27630 tablet_bootstrap.cc:654] T d4bcec8881bc4ce5a45cba85ca766552 P b9f8988d50b4448ea65bc250030db5bd: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:14.159777 27630 tablet_bootstrap.cc:492] T d4bcec8881bc4ce5a45cba85ca766552 P b9f8988d50b4448ea65bc250030db5bd: No bootstrap required, opened a new log
I20260619 14:52:14.159855 27630 ts_tablet_manager.cc:1397] T d4bcec8881bc4ce5a45cba85ca766552 P b9f8988d50b4448ea65bc250030db5bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:14.160240 27630 raft_consensus.cc:348] T d4bcec8881bc4ce5a45cba85ca766552 P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.160364 27630 raft_consensus.cc:374] T d4bcec8881bc4ce5a45cba85ca766552 P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:14.160398 27630 raft_consensus.cc:729] T d4bcec8881bc4ce5a45cba85ca766552 P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f8988d50b4448ea65bc250030db5bd, State: Initialized, Role: FOLLOWER
I20260619 14:52:14.160509 27630 consensus_queue.cc:260] T d4bcec8881bc4ce5a45cba85ca766552 P b9f8988d50b4448ea65bc250030db5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.160780 27630 ts_tablet_manager.cc:1428] T d4bcec8881bc4ce5a45cba85ca766552 P b9f8988d50b4448ea65bc250030db5bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:52:14.160962 27634 tablet_replica.cc:439] T d4bcec8881bc4ce5a45cba85ca766552 P b9f8988d50b4448ea65bc250030db5bd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } } }
I20260619 14:52:14.161108 27634 tablet_replica.cc:442] T d4bcec8881bc4ce5a45cba85ca766552 P b9f8988d50b4448ea65bc250030db5bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:52:14.230614 27649 raft_consensus.cc:482] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:52:14.230759 27649 raft_consensus.cc:504] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.231192 27649 leader_election.cc:288] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01932012b1bd4b8cb18f21da85bf1871 (127.19.34.66:43377), b9f8988d50b4448ea65bc250030db5bd (127.19.34.67:41781)
I20260619 14:52:14.231580 27412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4bcec8881bc4ce5a45cba85ca766552" candidate_uuid: "1cf2f87cb790496882ab5c0299489e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01932012b1bd4b8cb18f21da85bf1871" is_pre_election: true
I20260619 14:52:14.231653 27554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4bcec8881bc4ce5a45cba85ca766552" candidate_uuid: "1cf2f87cb790496882ab5c0299489e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9f8988d50b4448ea65bc250030db5bd" is_pre_election: true
I20260619 14:52:14.231741 27412 raft_consensus.cc:2454] T d4bcec8881bc4ce5a45cba85ca766552 P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cf2f87cb790496882ab5c0299489e92 in term 0.
I20260619 14:52:14.231814 27554 raft_consensus.cc:2454] T d4bcec8881bc4ce5a45cba85ca766552 P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cf2f87cb790496882ab5c0299489e92 in term 0.
I20260619 14:52:14.232077 27207 leader_election.cc:302] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01932012b1bd4b8cb18f21da85bf1871, 1cf2f87cb790496882ab5c0299489e92; no voters:
I20260619 14:52:14.232308 27649 raft_consensus.cc:2781] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:52:14.232403 27649 raft_consensus.cc:482] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:52:14.232447 27649 raft_consensus.cc:3037] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:14.233541 27649 raft_consensus.cc:504] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.233836 27649 leader_election.cc:288] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92 [CANDIDATE]: Term 1 election: Requested vote from peers 01932012b1bd4b8cb18f21da85bf1871 (127.19.34.66:43377), b9f8988d50b4448ea65bc250030db5bd (127.19.34.67:41781)
I20260619 14:52:14.234102 27412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4bcec8881bc4ce5a45cba85ca766552" candidate_uuid: "1cf2f87cb790496882ab5c0299489e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01932012b1bd4b8cb18f21da85bf1871"
I20260619 14:52:14.234207 27554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4bcec8881bc4ce5a45cba85ca766552" candidate_uuid: "1cf2f87cb790496882ab5c0299489e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9f8988d50b4448ea65bc250030db5bd"
I20260619 14:52:14.234246 27412 raft_consensus.cc:3037] T d4bcec8881bc4ce5a45cba85ca766552 P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:14.234306 27554 raft_consensus.cc:3037] T d4bcec8881bc4ce5a45cba85ca766552 P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:14.235266 27412 raft_consensus.cc:2454] T d4bcec8881bc4ce5a45cba85ca766552 P 01932012b1bd4b8cb18f21da85bf1871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf2f87cb790496882ab5c0299489e92 in term 1.
I20260619 14:52:14.235266 27554 raft_consensus.cc:2454] T d4bcec8881bc4ce5a45cba85ca766552 P b9f8988d50b4448ea65bc250030db5bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf2f87cb790496882ab5c0299489e92 in term 1.
I20260619 14:52:14.235534 27207 leader_election.cc:302] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01932012b1bd4b8cb18f21da85bf1871, 1cf2f87cb790496882ab5c0299489e92; no voters:
I20260619 14:52:14.235713 27649 raft_consensus.cc:2781] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:14.235803 27649 raft_consensus.cc:686] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92 [term 1 LEADER]: Becoming Leader. State: Replica: 1cf2f87cb790496882ab5c0299489e92, State: Running, Role: LEADER
I20260619 14:52:14.235931 27649 consensus_queue.cc:237] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.236665 27639 tablet_replica.cc:439] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92: TxnStatusTablet state changed. Reason: New leader 1cf2f87cb790496882ab5c0299489e92. Latest consensus state: current_term: 1 leader_uuid: "1cf2f87cb790496882ab5c0299489e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } } }
I20260619 14:52:14.236788 27639 tablet_replica.cc:442] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:52:14.237010 27672 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 14:52:14.237156 27672 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 14:52:14.237618 27115 catalog_manager.cc:5495] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cf2f87cb790496882ab5c0299489e92 (127.19.34.65). New cstate: current_term: 1 leader_uuid: "1cf2f87cb790496882ab5c0299489e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } health_report { overall_health: UNKNOWN } } }
I20260619 14:52:14.262143 27667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.258772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34119 (local address 127.0.0.1:44732)
0619 14:52:14.258892 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:14.258907 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:14.258999 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:14.259264 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:14.259266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:14.259272 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:52:14.259362 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:14.259366 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.260277 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.260285 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:14.260843 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:14.260850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.261425 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.261428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:14.261488 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.261525 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:52:14.261907 (+ 382us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:52:14.261917 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 14:52:14.261963 (+ 46us) client_negotiation.cc:240] Negotiation successful
0619 14:52:14.262026 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260619 14:52:14.262173 27627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.258833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44732 (local address 127.19.34.65:34119)
0619 14:52:14.258931 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:14.258935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:14.258948 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:52:14.259115 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:14.259118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:14.259163 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:14.259245 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:52:14.259462 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.260181 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.260953 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.261322 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.261422 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.261717 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:52:14.261801 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:52:14.261886 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:14.261979 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 14:52:14.262033 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:52:14.264226 27554 raft_consensus.cc:1261] T d4bcec8881bc4ce5a45cba85ca766552 P b9f8988d50b4448ea65bc250030db5bd [term 1 FOLLOWER]: Refusing update from remote peer 1cf2f87cb790496882ab5c0299489e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:14.264245 27412 raft_consensus.cc:1261] T d4bcec8881bc4ce5a45cba85ca766552 P 01932012b1bd4b8cb18f21da85bf1871 [term 1 FOLLOWER]: Refusing update from remote peer 1cf2f87cb790496882ab5c0299489e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:14.264595 27639 consensus_queue.cc:1035] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:14.264736 27649 consensus_queue.cc:1035] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:14.266939 27634 tablet_replica.cc:439] T d4bcec8881bc4ce5a45cba85ca766552 P b9f8988d50b4448ea65bc250030db5bd: TxnStatusTablet state changed. Reason: New leader 1cf2f87cb790496882ab5c0299489e92. Latest consensus state: current_term: 1 leader_uuid: "1cf2f87cb790496882ab5c0299489e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } } }
I20260619 14:52:14.267083 27634 tablet_replica.cc:442] T d4bcec8881bc4ce5a45cba85ca766552 P b9f8988d50b4448ea65bc250030db5bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:52:14.267442 27658 tablet_replica.cc:439] T d4bcec8881bc4ce5a45cba85ca766552 P 01932012b1bd4b8cb18f21da85bf1871: TxnStatusTablet state changed. Reason: New leader 1cf2f87cb790496882ab5c0299489e92. Latest consensus state: current_term: 1 leader_uuid: "1cf2f87cb790496882ab5c0299489e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } } }
I20260619 14:52:14.267556 27658 tablet_replica.cc:442] T d4bcec8881bc4ce5a45cba85ca766552 P 01932012b1bd4b8cb18f21da85bf1871: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:52:14.268448 27639 tablet_replica.cc:439] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cf2f87cb790496882ab5c0299489e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } } }
I20260619 14:52:14.268556 27639 tablet_replica.cc:442] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:52:14.270176 27634 tablet_replica.cc:439] T d4bcec8881bc4ce5a45cba85ca766552 P b9f8988d50b4448ea65bc250030db5bd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cf2f87cb790496882ab5c0299489e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } } }
I20260619 14:52:14.270256 27634 tablet_replica.cc:442] T d4bcec8881bc4ce5a45cba85ca766552 P b9f8988d50b4448ea65bc250030db5bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:52:14.270761 27644 mvcc.cc:205] Tried to move back new op lower bound from 7298583487542616064 to 7298583487431905280. Current Snapshot: MvccSnapshot[applied={T|T < 7298583487542616064 or (T in {7298583487542616064})}]
I20260619 14:52:14.271082 27649 tablet_replica.cc:439] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cf2f87cb790496882ab5c0299489e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } } }
I20260619 14:52:14.271173 27649 tablet_replica.cc:442] T d4bcec8881bc4ce5a45cba85ca766552 P 1cf2f87cb790496882ab5c0299489e92: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:52:14.271867 27658 tablet_replica.cc:439] T d4bcec8881bc4ce5a45cba85ca766552 P 01932012b1bd4b8cb18f21da85bf1871: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cf2f87cb790496882ab5c0299489e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } } }
I20260619 14:52:14.272014 27658 tablet_replica.cc:442] T d4bcec8881bc4ce5a45cba85ca766552 P 01932012b1bd4b8cb18f21da85bf1871: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:52:14.277237 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.273562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.19.34.66:43377)
0619 14:52:14.273691 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:14.273695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:14.273710 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:52:14.273780 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:14.273782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:14.273841 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:14.273962 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:52:14.274285 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.274944 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.275846 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.276073 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.276187 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.276512 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:52:14.276907 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:52:14.277020 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:14.277071 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 14:52:14.277126 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:52:14.287551 27627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.283468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:39069 (local address 127.19.34.65:34119)
0619 14:52:14.283599 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:14.283604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:14.283826 (+ 222us) server_negotiation.cc:407] Connection header received
0619 14:52:14.283946 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:14.283948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:14.283984 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:14.284042 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:52:14.284357 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.284803 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.286554 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.286809 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.286880 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.287207 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:52:14.287259 (+ 52us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:52:14.287310 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:14.287431 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 14:52:14.287470 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:52:14.287654 27678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.283368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34119 (local address 127.19.34.66:39069)
0619 14:52:14.283738 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:14.283754 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:14.283859 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:14.284083 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:14.284086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:14.284093 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:52:14.284210 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:14.284217 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.284924 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.284928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:14.286424 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:14.286436 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.286919 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.286923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:14.287029 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.287087 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:52:14.287339 (+ 252us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:52:14.287346 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:52:14.287418 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 14:52:14.287493 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":128,"threads_started":1}
I20260619 14:52:14.293460 27554 raft_consensus.cc:1261] T b78fb165207b43238a3c0b616456c7d7 P b9f8988d50b4448ea65bc250030db5bd [term 1 FOLLOWER]: Refusing update from remote peer 01932012b1bd4b8cb18f21da85bf1871: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:14.293586 27274 raft_consensus.cc:1261] T b78fb165207b43238a3c0b616456c7d7 P 1cf2f87cb790496882ab5c0299489e92 [term 1 FOLLOWER]: Refusing update from remote peer 01932012b1bd4b8cb18f21da85bf1871: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:14.293892 27658 consensus_queue.cc:1035] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:14.294015 27635 consensus_queue.cc:1035] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:14.321216 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.317377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:38539 (local address 127.19.34.66:43377)
0619 14:52:14.317475 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:14.317481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:14.317498 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:52:14.317539 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:14.317543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:14.317587 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:14.317660 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:52:14.317958 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.318687 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.319899 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.320225 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.320332 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.320723 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:52:14.320799 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:52:14.320854 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:14.321031 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 14:52:14.321091 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 14:52:14.321758 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.317073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:43377 (local address 127.19.34.65:38539)
0619 14:52:14.317357 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:14.317372 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:14.317457 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:14.317695 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:14.317699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:14.317708 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:52:14.317823 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:14.317829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.318837 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.318841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:14.319754 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:14.319765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.320443 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.320447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:14.320530 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.320576 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:52:14.320929 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:52:14.320935 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:52:14.321543 (+ 608us) client_negotiation.cc:240] Negotiation successful
0619 14:52:14.321621 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":95,"threads_started":1}
I20260619 14:52:14.346228 27115 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:14.346371 27115 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:14.348517 27115 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16: Sending DeleteTablet for 3 replicas of tablet b78fb165207b43238a3c0b616456c7d7
I20260619 14:52:14.348982 27254 tablet_service.cc:1514] Processing DeleteTablet for tablet b78fb165207b43238a3c0b616456c7d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:14 UTC) from {username='oryx', principal='oryx/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:44702
I20260619 14:52:14.349032 27394 tablet_service.cc:1514] Processing DeleteTablet for tablet b78fb165207b43238a3c0b616456c7d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:14 UTC) from {username='oryx', principal='oryx/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:33426
I20260619 14:52:14.349151 27654 tablet_replica.cc:330] T b78fb165207b43238a3c0b616456c7d7 P 1cf2f87cb790496882ab5c0299489e92: stopping tablet replica
I20260619 14:52:14.349251 27654 raft_consensus.cc:2229] T b78fb165207b43238a3c0b616456c7d7 P 1cf2f87cb790496882ab5c0299489e92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:14.349344 27654 raft_consensus.cc:2258] T b78fb165207b43238a3c0b616456c7d7 P 1cf2f87cb790496882ab5c0299489e92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:14.349498 27652 tablet_replica.cc:330] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871: stopping tablet replica
I20260619 14:52:14.349594 27652 raft_consensus.cc:2229] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:52:14.349853 27652 raft_consensus.cc:2258] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:14.350287 27534 tablet_service.cc:1514] Processing DeleteTablet for tablet b78fb165207b43238a3c0b616456c7d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:14 UTC) from {username='oryx', principal='oryx/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:59850
I20260619 14:52:14.350436 27653 tablet_replica.cc:330] T b78fb165207b43238a3c0b616456c7d7 P b9f8988d50b4448ea65bc250030db5bd: stopping tablet replica
I20260619 14:52:14.350528 27653 raft_consensus.cc:2229] T b78fb165207b43238a3c0b616456c7d7 P b9f8988d50b4448ea65bc250030db5bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:14.350522 27652 ts_tablet_manager.cc:1905] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:14.350625 27653 raft_consensus.cc:2258] T b78fb165207b43238a3c0b616456c7d7 P b9f8988d50b4448ea65bc250030db5bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:14.351740 27653 ts_tablet_manager.cc:1905] T b78fb165207b43238a3c0b616456c7d7 P b9f8988d50b4448ea65bc250030db5bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:14.353227 27652 ts_tablet_manager.cc:1918] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:52:14.353286 27652 log.cc:1196] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-1/wal/wals/b78fb165207b43238a3c0b616456c7d7
I20260619 14:52:14.353511 27652 ts_tablet_manager.cc:1939] T b78fb165207b43238a3c0b616456c7d7 P 01932012b1bd4b8cb18f21da85bf1871: Deleting consensus metadata
I20260619 14:52:14.354043 27103 catalog_manager.cc:4841] TS 01932012b1bd4b8cb18f21da85bf1871 (127.19.34.66:43377): tablet b78fb165207b43238a3c0b616456c7d7 (table test-table [id=0c09855a47ee4d19883c2e747f3f69d1]) successfully deleted
I20260619 14:52:14.354293 27653 ts_tablet_manager.cc:1918] T b78fb165207b43238a3c0b616456c7d7 P b9f8988d50b4448ea65bc250030db5bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:52:14.354357 27653 log.cc:1196] T b78fb165207b43238a3c0b616456c7d7 P b9f8988d50b4448ea65bc250030db5bd: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/b78fb165207b43238a3c0b616456c7d7
I20260619 14:52:14.354595 27653 ts_tablet_manager.cc:1939] T b78fb165207b43238a3c0b616456c7d7 P b9f8988d50b4448ea65bc250030db5bd: Deleting consensus metadata
I20260619 14:52:14.355198 27101 catalog_manager.cc:4841] TS b9f8988d50b4448ea65bc250030db5bd (127.19.34.67:41781): tablet b78fb165207b43238a3c0b616456c7d7 (table test-table [id=0c09855a47ee4d19883c2e747f3f69d1]) successfully deleted
I20260619 14:52:14.355659 27654 ts_tablet_manager.cc:1905] T b78fb165207b43238a3c0b616456c7d7 P 1cf2f87cb790496882ab5c0299489e92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:14.358227 27654 ts_tablet_manager.cc:1918] T b78fb165207b43238a3c0b616456c7d7 P 1cf2f87cb790496882ab5c0299489e92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:52:14.358296 27654 log.cc:1196] T b78fb165207b43238a3c0b616456c7d7 P 1cf2f87cb790496882ab5c0299489e92: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-0/wal/wals/b78fb165207b43238a3c0b616456c7d7
I20260619 14:52:14.358566 27654 ts_tablet_manager.cc:1939] T b78fb165207b43238a3c0b616456c7d7 P 1cf2f87cb790496882ab5c0299489e92: Deleting consensus metadata
I20260619 14:52:14.359167 27104 catalog_manager.cc:4841] TS 1cf2f87cb790496882ab5c0299489e92 (127.19.34.65:34119): tablet b78fb165207b43238a3c0b616456c7d7 (table test-table [id=0c09855a47ee4d19883c2e747f3f69d1]) successfully deleted
Jun 19 14:52:14 dist-test-slave-3qdh krb5kdc[27072](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:52:14 dist-test-slave-3qdh krb5kdc[27072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:14 dist-test-slave-3qdh krb5kdc[27072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:52:14.390074 27166 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:14.382644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35962 (local address 127.19.34.126:43059)
0619 14:52:14.382765 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:14.382770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:14.382788 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:52:14.382903 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:14.382906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:14.382953 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:14.383046 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:14.383856 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.384630 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.385863 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.386223 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.386340 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.389927 (+ 3587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35962: BlockingRecv error: recv got EOF from 127.0.0.1:35962 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 14:52:14 dist-test-slave-3qdh krb5kdc[27072](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM
I20260619 14:52:14.401906 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.391978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35966 (local address 127.19.34.126:43059)
0619 14:52:14.392143 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:14.392150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:14.392464 (+ 314us) server_negotiation.cc:407] Connection header received
0619 14:52:14.392558 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:14.392561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:14.392611 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:14.392703 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:14.393372 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.394162 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.395284 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.395649 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.395762 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.398485 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:14.398489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:14.398491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:14.398515 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:14.400148 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:14.400649 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:14.400653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:14.400655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:14.400737 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:14.401036 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:14.401038 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:14.401040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:14.401235 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:14.401351 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:14.401623 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 14:52:14.401738 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:52:14.404559 27115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:52:14.404985 27115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:52:14.409894 27394 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1c50bc521d40dc99203048b6928912 (DEFAULT_TABLE table=test-table [id=815611fcc2f641c88a76d9ce668693a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:14.409894 27254 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1c50bc521d40dc99203048b6928912 (DEFAULT_TABLE table=test-table [id=815611fcc2f641c88a76d9ce668693a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:14.410195 27254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1c50bc521d40dc99203048b6928912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:14.410195 27394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1c50bc521d40dc99203048b6928912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:14.412647 27628 tablet_bootstrap.cc:492] T 2b1c50bc521d40dc99203048b6928912 P 01932012b1bd4b8cb18f21da85bf1871: Bootstrap starting.
I20260619 14:52:14.413458 27628 tablet_bootstrap.cc:654] T 2b1c50bc521d40dc99203048b6928912 P 01932012b1bd4b8cb18f21da85bf1871: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:14.414182 27534 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1c50bc521d40dc99203048b6928912 (DEFAULT_TABLE table=test-table [id=815611fcc2f641c88a76d9ce668693a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:14.414485 27534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1c50bc521d40dc99203048b6928912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:14.414646 27628 tablet_bootstrap.cc:492] T 2b1c50bc521d40dc99203048b6928912 P 01932012b1bd4b8cb18f21da85bf1871: No bootstrap required, opened a new log
I20260619 14:52:14.414758 27628 ts_tablet_manager.cc:1397] T 2b1c50bc521d40dc99203048b6928912 P 01932012b1bd4b8cb18f21da85bf1871: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:52:14.415213 27628 raft_consensus.cc:348] T 2b1c50bc521d40dc99203048b6928912 P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.415352 27628 raft_consensus.cc:374] T 2b1c50bc521d40dc99203048b6928912 P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:14.415395 27628 raft_consensus.cc:729] T 2b1c50bc521d40dc99203048b6928912 P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01932012b1bd4b8cb18f21da85bf1871, State: Initialized, Role: FOLLOWER
I20260619 14:52:14.415545 27628 consensus_queue.cc:260] T 2b1c50bc521d40dc99203048b6928912 P 01932012b1bd4b8cb18f21da85bf1871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.415874 27628 ts_tablet_manager.cc:1428] T 2b1c50bc521d40dc99203048b6928912 P 01932012b1bd4b8cb18f21da85bf1871: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:52:14.416941 27629 tablet_bootstrap.cc:492] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92: Bootstrap starting.
I20260619 14:52:14.417002 27630 tablet_bootstrap.cc:492] T 2b1c50bc521d40dc99203048b6928912 P b9f8988d50b4448ea65bc250030db5bd: Bootstrap starting.
I20260619 14:52:14.417580 27629 tablet_bootstrap.cc:654] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:14.417991 27630 tablet_bootstrap.cc:654] T 2b1c50bc521d40dc99203048b6928912 P b9f8988d50b4448ea65bc250030db5bd: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:14.418696 27629 tablet_bootstrap.cc:492] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92: No bootstrap required, opened a new log
I20260619 14:52:14.418780 27629 ts_tablet_manager.cc:1397] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:52:14.419122 27629 raft_consensus.cc:348] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.419232 27629 raft_consensus.cc:374] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:14.419277 27629 raft_consensus.cc:729] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf2f87cb790496882ab5c0299489e92, State: Initialized, Role: FOLLOWER
I20260619 14:52:14.419301 27630 tablet_bootstrap.cc:492] T 2b1c50bc521d40dc99203048b6928912 P b9f8988d50b4448ea65bc250030db5bd: No bootstrap required, opened a new log
I20260619 14:52:14.419394 27630 ts_tablet_manager.cc:1397] T 2b1c50bc521d40dc99203048b6928912 P b9f8988d50b4448ea65bc250030db5bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:14.419399 27629 consensus_queue.cc:260] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.419687 27629 ts_tablet_manager.cc:1428] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:52:14.419862 27630 raft_consensus.cc:348] T 2b1c50bc521d40dc99203048b6928912 P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.419983 27630 raft_consensus.cc:374] T 2b1c50bc521d40dc99203048b6928912 P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:14.420020 27630 raft_consensus.cc:729] T 2b1c50bc521d40dc99203048b6928912 P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f8988d50b4448ea65bc250030db5bd, State: Initialized, Role: FOLLOWER
I20260619 14:52:14.420174 27630 consensus_queue.cc:260] T 2b1c50bc521d40dc99203048b6928912 P b9f8988d50b4448ea65bc250030db5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.420864 27630 ts_tablet_manager.cc:1428] T 2b1c50bc521d40dc99203048b6928912 P b9f8988d50b4448ea65bc250030db5bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:52:14.615257 27636 raft_consensus.cc:482] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:52:14.615368 27636 raft_consensus.cc:504] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.615811 27636 leader_election.cc:288] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01932012b1bd4b8cb18f21da85bf1871 (127.19.34.66:43377), b9f8988d50b4448ea65bc250030db5bd (127.19.34.67:41781)
I20260619 14:52:14.616218 27412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1c50bc521d40dc99203048b6928912" candidate_uuid: "1cf2f87cb790496882ab5c0299489e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01932012b1bd4b8cb18f21da85bf1871" is_pre_election: true
I20260619 14:52:14.616218 27554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1c50bc521d40dc99203048b6928912" candidate_uuid: "1cf2f87cb790496882ab5c0299489e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9f8988d50b4448ea65bc250030db5bd" is_pre_election: true
I20260619 14:52:14.616384 27412 raft_consensus.cc:2454] T 2b1c50bc521d40dc99203048b6928912 P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cf2f87cb790496882ab5c0299489e92 in term 0.
I20260619 14:52:14.616384 27554 raft_consensus.cc:2454] T 2b1c50bc521d40dc99203048b6928912 P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cf2f87cb790496882ab5c0299489e92 in term 0.
I20260619 14:52:14.616722 27205 leader_election.cc:302] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cf2f87cb790496882ab5c0299489e92, b9f8988d50b4448ea65bc250030db5bd; no voters:
I20260619 14:52:14.616919 27636 raft_consensus.cc:2781] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:52:14.616986 27636 raft_consensus.cc:482] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:52:14.617035 27636 raft_consensus.cc:3037] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:14.617977 27636 raft_consensus.cc:504] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.618278 27636 leader_election.cc:288] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 [CANDIDATE]: Term 1 election: Requested vote from peers 01932012b1bd4b8cb18f21da85bf1871 (127.19.34.66:43377), b9f8988d50b4448ea65bc250030db5bd (127.19.34.67:41781)
I20260619 14:52:14.618637 27554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1c50bc521d40dc99203048b6928912" candidate_uuid: "1cf2f87cb790496882ab5c0299489e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9f8988d50b4448ea65bc250030db5bd"
I20260619 14:52:14.618634 27412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1c50bc521d40dc99203048b6928912" candidate_uuid: "1cf2f87cb790496882ab5c0299489e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01932012b1bd4b8cb18f21da85bf1871"
I20260619 14:52:14.618793 27554 raft_consensus.cc:3037] T 2b1c50bc521d40dc99203048b6928912 P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:14.618793 27412 raft_consensus.cc:3037] T 2b1c50bc521d40dc99203048b6928912 P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:14.619820 27554 raft_consensus.cc:2454] T 2b1c50bc521d40dc99203048b6928912 P b9f8988d50b4448ea65bc250030db5bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf2f87cb790496882ab5c0299489e92 in term 1.
I20260619 14:52:14.619822 27412 raft_consensus.cc:2454] T 2b1c50bc521d40dc99203048b6928912 P 01932012b1bd4b8cb18f21da85bf1871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf2f87cb790496882ab5c0299489e92 in term 1.
I20260619 14:52:14.620116 27207 leader_election.cc:302] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01932012b1bd4b8cb18f21da85bf1871, 1cf2f87cb790496882ab5c0299489e92; no voters:
I20260619 14:52:14.620353 27636 raft_consensus.cc:2781] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:14.620443 27636 raft_consensus.cc:686] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 [term 1 LEADER]: Becoming Leader. State: Replica: 1cf2f87cb790496882ab5c0299489e92, State: Running, Role: LEADER
I20260619 14:52:14.620572 27636 consensus_queue.cc:237] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.621940 27115 catalog_manager.cc:5495] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cf2f87cb790496882ab5c0299489e92 (127.19.34.65). New cstate: current_term: 1 leader_uuid: "1cf2f87cb790496882ab5c0299489e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } health_report { overall_health: UNKNOWN } } }
I20260619 14:52:14.648048 27627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.645213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44744 (local address 127.19.34.65:34119)
0619 14:52:14.645346 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:14.645349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:14.645362 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:52:14.645473 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:14.645475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:14.645532 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:14.645605 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:52:14.645866 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.646414 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.647173 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.647379 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.647449 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.647653 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:52:14.647709 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:52:14.647759 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:14.647854 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 14:52:14.647890 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:52:14.649647 27412 raft_consensus.cc:1261] T 2b1c50bc521d40dc99203048b6928912 P 01932012b1bd4b8cb18f21da85bf1871 [term 1 FOLLOWER]: Refusing update from remote peer 1cf2f87cb790496882ab5c0299489e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:14.649647 27554 raft_consensus.cc:1261] T 2b1c50bc521d40dc99203048b6928912 P b9f8988d50b4448ea65bc250030db5bd [term 1 FOLLOWER]: Refusing update from remote peer 1cf2f87cb790496882ab5c0299489e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:14.649989 27639 consensus_queue.cc:1035] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:14.650101 27636 consensus_queue.cc:1035] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:14.656949 27117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:14.657094 27117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:14.658655 27117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16: Sending DeleteTablet for 3 replicas of tablet 2b1c50bc521d40dc99203048b6928912
I20260619 14:52:14.659135 27254 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1c50bc521d40dc99203048b6928912 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:14 UTC) from {username='oryx', principal='oryx/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:44702
I20260619 14:52:14.659194 27394 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1c50bc521d40dc99203048b6928912 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:14 UTC) from {username='oryx', principal='oryx/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:33426
I20260619 14:52:14.659201 27534 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1c50bc521d40dc99203048b6928912 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:14 UTC) from {username='oryx', principal='oryx/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:59850
I20260619 14:52:14.659283 27654 tablet_replica.cc:330] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92: stopping tablet replica
I20260619 14:52:14.659330 27653 tablet_replica.cc:330] T 2b1c50bc521d40dc99203048b6928912 P b9f8988d50b4448ea65bc250030db5bd: stopping tablet replica
I20260619 14:52:14.659389 27654 raft_consensus.cc:2229] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:52:14.659416 27653 raft_consensus.cc:2229] T 2b1c50bc521d40dc99203048b6928912 P b9f8988d50b4448ea65bc250030db5bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:14.659526 27653 raft_consensus.cc:2258] T 2b1c50bc521d40dc99203048b6928912 P b9f8988d50b4448ea65bc250030db5bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:14.659555 27654 raft_consensus.cc:2258] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:14.659730 27652 tablet_replica.cc:330] T 2b1c50bc521d40dc99203048b6928912 P 01932012b1bd4b8cb18f21da85bf1871: stopping tablet replica
I20260619 14:52:14.659857 27652 raft_consensus.cc:2229] T 2b1c50bc521d40dc99203048b6928912 P 01932012b1bd4b8cb18f21da85bf1871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:14.659974 27652 raft_consensus.cc:2258] T 2b1c50bc521d40dc99203048b6928912 P 01932012b1bd4b8cb18f21da85bf1871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:14.659904 27117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:52:14.660225 27117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:52:14.660262 27654 ts_tablet_manager.cc:1905] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:14.660647 27653 ts_tablet_manager.cc:1905] T 2b1c50bc521d40dc99203048b6928912 P b9f8988d50b4448ea65bc250030db5bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:14.660643 27652 ts_tablet_manager.cc:1905] T 2b1c50bc521d40dc99203048b6928912 P 01932012b1bd4b8cb18f21da85bf1871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:14.662245 27654 ts_tablet_manager.cc:1918] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:14.662290 27654 log.cc:1196] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-0/wal/wals/2b1c50bc521d40dc99203048b6928912
I20260619 14:52:14.662454 27654 ts_tablet_manager.cc:1939] T 2b1c50bc521d40dc99203048b6928912 P 1cf2f87cb790496882ab5c0299489e92: Deleting consensus metadata
I20260619 14:52:14.662741 27652 ts_tablet_manager.cc:1918] T 2b1c50bc521d40dc99203048b6928912 P 01932012b1bd4b8cb18f21da85bf1871: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:14.662741 27653 ts_tablet_manager.cc:1918] T 2b1c50bc521d40dc99203048b6928912 P b9f8988d50b4448ea65bc250030db5bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:14.662814 27652 log.cc:1196] T 2b1c50bc521d40dc99203048b6928912 P 01932012b1bd4b8cb18f21da85bf1871: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-1/wal/wals/2b1c50bc521d40dc99203048b6928912
I20260619 14:52:14.662814 27653 log.cc:1196] T 2b1c50bc521d40dc99203048b6928912 P b9f8988d50b4448ea65bc250030db5bd: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/2b1c50bc521d40dc99203048b6928912
I20260619 14:52:14.663043 27653 ts_tablet_manager.cc:1939] T 2b1c50bc521d40dc99203048b6928912 P b9f8988d50b4448ea65bc250030db5bd: Deleting consensus metadata
I20260619 14:52:14.663043 27652 ts_tablet_manager.cc:1939] T 2b1c50bc521d40dc99203048b6928912 P 01932012b1bd4b8cb18f21da85bf1871: Deleting consensus metadata
I20260619 14:52:14.663014 27104 catalog_manager.cc:4841] TS 1cf2f87cb790496882ab5c0299489e92 (127.19.34.65:34119): tablet 2b1c50bc521d40dc99203048b6928912 (table test-table [id=815611fcc2f641c88a76d9ce668693a8]) successfully deleted
I20260619 14:52:14.663578 27101 catalog_manager.cc:4841] TS b9f8988d50b4448ea65bc250030db5bd (127.19.34.67:41781): tablet 2b1c50bc521d40dc99203048b6928912 (table test-table [id=815611fcc2f641c88a76d9ce668693a8]) successfully deleted
I20260619 14:52:14.663849 27103 catalog_manager.cc:4841] TS 01932012b1bd4b8cb18f21da85bf1871 (127.19.34.66:43377): tablet 2b1c50bc521d40dc99203048b6928912 (table test-table [id=815611fcc2f641c88a76d9ce668693a8]) successfully deleted
I20260619 14:52:14.665663 27254 tablet_service.cc:1467] Processing CreateTablet for tablet 9cba96f7d7ae41c3bc190ac1f8758b6d (DEFAULT_TABLE table=test-table [id=d2e2be58bd5d4eb48083e5835e6be86e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:14.665912 27254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cba96f7d7ae41c3bc190ac1f8758b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:14.665939 27534 tablet_service.cc:1467] Processing CreateTablet for tablet 9cba96f7d7ae41c3bc190ac1f8758b6d (DEFAULT_TABLE table=test-table [id=d2e2be58bd5d4eb48083e5835e6be86e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:14.666128 27534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cba96f7d7ae41c3bc190ac1f8758b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:14.667903 27629 tablet_bootstrap.cc:492] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92: Bootstrap starting.
I20260619 14:52:14.668031 27630 tablet_bootstrap.cc:492] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd: Bootstrap starting.
I20260619 14:52:14.668285 27394 tablet_service.cc:1467] Processing CreateTablet for tablet 9cba96f7d7ae41c3bc190ac1f8758b6d (DEFAULT_TABLE table=test-table [id=d2e2be58bd5d4eb48083e5835e6be86e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:14.668541 27394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cba96f7d7ae41c3bc190ac1f8758b6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:14.668797 27629 tablet_bootstrap.cc:654] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:14.669377 27630 tablet_bootstrap.cc:654] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:14.669816 27629 tablet_bootstrap.cc:492] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92: No bootstrap required, opened a new log
I20260619 14:52:14.669898 27629 ts_tablet_manager.cc:1397] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:14.670128 27628 tablet_bootstrap.cc:492] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 01932012b1bd4b8cb18f21da85bf1871: Bootstrap starting.
I20260619 14:52:14.670243 27629 raft_consensus.cc:348] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.670338 27629 raft_consensus.cc:374] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:14.670375 27629 raft_consensus.cc:729] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf2f87cb790496882ab5c0299489e92, State: Initialized, Role: FOLLOWER
I20260619 14:52:14.670480 27629 consensus_queue.cc:260] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.670703 27629 ts_tablet_manager.cc:1428] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:52:14.670838 27628 tablet_bootstrap.cc:654] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 01932012b1bd4b8cb18f21da85bf1871: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:14.671545 27630 tablet_bootstrap.cc:492] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd: No bootstrap required, opened a new log
I20260619 14:52:14.671634 27630 ts_tablet_manager.cc:1397] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 14:52:14.672051 27630 raft_consensus.cc:348] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.672147 27630 raft_consensus.cc:374] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:14.672187 27630 raft_consensus.cc:729] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9f8988d50b4448ea65bc250030db5bd, State: Initialized, Role: FOLLOWER
I20260619 14:52:14.672322 27630 consensus_queue.cc:260] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.672537 27630 ts_tablet_manager.cc:1428] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:52:14.673008 27628 tablet_bootstrap.cc:492] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 01932012b1bd4b8cb18f21da85bf1871: No bootstrap required, opened a new log
I20260619 14:52:14.673105 27628 ts_tablet_manager.cc:1397] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 01932012b1bd4b8cb18f21da85bf1871: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 14:52:14.673461 27628 raft_consensus.cc:348] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.673552 27628 raft_consensus.cc:374] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:14.673573 27628 raft_consensus.cc:729] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01932012b1bd4b8cb18f21da85bf1871, State: Initialized, Role: FOLLOWER
I20260619 14:52:14.673708 27628 consensus_queue.cc:260] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 01932012b1bd4b8cb18f21da85bf1871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.673936 27628 ts_tablet_manager.cc:1428] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 01932012b1bd4b8cb18f21da85bf1871: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:52:14.853008 27634 raft_consensus.cc:482] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:52:14.853123 27634 raft_consensus.cc:504] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.854097 27634 leader_election.cc:288] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cf2f87cb790496882ab5c0299489e92 (127.19.34.65:34119), 01932012b1bd4b8cb18f21da85bf1871 (127.19.34.66:43377)
I20260619 14:52:14.857923 27636 raft_consensus.cc:482] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:52:14.858039 27636 raft_consensus.cc:504] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.858371 27636 leader_election.cc:288] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01932012b1bd4b8cb18f21da85bf1871 (127.19.34.66:43377), b9f8988d50b4448ea65bc250030db5bd (127.19.34.67:41781)
I20260619 14:52:14.858714 27412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cba96f7d7ae41c3bc190ac1f8758b6d" candidate_uuid: "1cf2f87cb790496882ab5c0299489e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01932012b1bd4b8cb18f21da85bf1871" is_pre_election: true
I20260619 14:52:14.858845 27412 raft_consensus.cc:2454] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cf2f87cb790496882ab5c0299489e92 in term 0.
I20260619 14:52:14.859143 27207 leader_election.cc:302] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01932012b1bd4b8cb18f21da85bf1871, 1cf2f87cb790496882ab5c0299489e92; no voters:
I20260619 14:52:14.859318 27636 raft_consensus.cc:2781] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:52:14.859274 27554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cba96f7d7ae41c3bc190ac1f8758b6d" candidate_uuid: "1cf2f87cb790496882ab5c0299489e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9f8988d50b4448ea65bc250030db5bd" is_pre_election: true
I20260619 14:52:14.859359 27636 raft_consensus.cc:482] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:52:14.859395 27636 raft_consensus.cc:3037] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:14.859396 27554 raft_consensus.cc:2454] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cf2f87cb790496882ab5c0299489e92 in term 0.
I20260619 14:52:14.859439 27709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.854326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:43377 (local address 127.19.34.67:51277)
0619 14:52:14.854662 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:14.854677 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:14.854779 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:14.855058 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:14.855061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:14.855084 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:14.855322 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:14.855328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.856456 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.856460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:14.858034 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:14.858044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.859012 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.859016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:14.859087 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.859105 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:52:14.859186 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:52:14.859305 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":336,"spinlock_wait_cycles":4352,"thread_start_us":57,"threads_started":1}
I20260619 14:52:14.859464 27710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.854413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:51277 (local address 127.19.34.66:43377)
0619 14:52:14.854682 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:14.854685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:14.854751 (+ 66us) server_negotiation.cc:407] Connection header received
0619 14:52:14.854899 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:14.854902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:14.854950 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:14.855034 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:14.855475 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.856333 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.858179 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.858796 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.858903 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.858956 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:14.859273 (+ 317us) server_negotiation.cc:299] Negotiation successful
0619 14:52:14.859352 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":65,"threads_started":1}
I20260619 14:52:14.860201 27708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.854326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34119 (local address 127.19.34.67:46347)
0619 14:52:14.854606 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:14.854621 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:14.854779 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:14.855058 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:14.855061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:14.855084 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:14.855322 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:14.855328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.856456 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.856460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:14.858034 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:14.858044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.858795 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.858798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:14.858894 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.858918 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:52:14.859164 (+ 246us) client_negotiation.cc:240] Negotiation successful
0619 14:52:14.859259 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":179,"spinlock_wait_cycles":25856,"thread_start_us":75,"threads_started":1}
I20260619 14:52:14.860641 27636 raft_consensus.cc:504] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.860965 27636 leader_election.cc:288] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [CANDIDATE]: Term 1 election: Requested vote from peers 01932012b1bd4b8cb18f21da85bf1871 (127.19.34.66:43377), b9f8988d50b4448ea65bc250030db5bd (127.19.34.67:41781)
I20260619 14:52:14.861171 27412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cba96f7d7ae41c3bc190ac1f8758b6d" candidate_uuid: "b9f8988d50b4448ea65bc250030db5bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01932012b1bd4b8cb18f21da85bf1871" is_pre_election: true
I20260619 14:52:14.861263 27554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cba96f7d7ae41c3bc190ac1f8758b6d" candidate_uuid: "1cf2f87cb790496882ab5c0299489e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9f8988d50b4448ea65bc250030db5bd"
I20260619 14:52:14.861299 27412 raft_consensus.cc:2454] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9f8988d50b4448ea65bc250030db5bd in term 0.
I20260619 14:52:14.861302 27414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cba96f7d7ae41c3bc190ac1f8758b6d" candidate_uuid: "1cf2f87cb790496882ab5c0299489e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01932012b1bd4b8cb18f21da85bf1871"
I20260619 14:52:14.861373 27554 raft_consensus.cc:3037] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:14.861413 27414 raft_consensus.cc:3037] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 01932012b1bd4b8cb18f21da85bf1871 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:14.861702 27487 leader_election.cc:302] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01932012b1bd4b8cb18f21da85bf1871, b9f8988d50b4448ea65bc250030db5bd; no voters:
I20260619 14:52:14.861990 27627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.854413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:46347 (local address 127.19.34.65:34119)
0619 14:52:14.854617 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:14.854623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:14.854730 (+ 107us) server_negotiation.cc:407] Connection header received
0619 14:52:14.854899 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:14.854902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:14.854950 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:14.855033 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:14.855599 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.856348 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.858139 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.858673 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.858790 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.858844 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:14.859036 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 14:52:14.859103 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260619 14:52:14.862330 27274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cba96f7d7ae41c3bc190ac1f8758b6d" candidate_uuid: "b9f8988d50b4448ea65bc250030db5bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cf2f87cb790496882ab5c0299489e92" is_pre_election: true
I20260619 14:52:14.862416 27554 raft_consensus.cc:2454] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf2f87cb790496882ab5c0299489e92 in term 1.
I20260619 14:52:14.862491 27274 raft_consensus.cc:2379] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9f8988d50b4448ea65bc250030db5bd in current term 1: Already voted for candidate 1cf2f87cb790496882ab5c0299489e92 in this term.
I20260619 14:52:14.862540 27634 raft_consensus.cc:2741] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260619 14:52:14.862578 27414 raft_consensus.cc:2454] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 01932012b1bd4b8cb18f21da85bf1871 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cf2f87cb790496882ab5c0299489e92 in term 1.
I20260619 14:52:14.862730 27205 leader_election.cc:302] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cf2f87cb790496882ab5c0299489e92, b9f8988d50b4448ea65bc250030db5bd; no voters:
I20260619 14:52:14.862859 27636 raft_consensus.cc:2781] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:14.862993 27636 raft_consensus.cc:686] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [term 1 LEADER]: Becoming Leader. State: Replica: 1cf2f87cb790496882ab5c0299489e92, State: Running, Role: LEADER
I20260619 14:52:14.863145 27636 consensus_queue.cc:237] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } }
I20260619 14:52:14.864634 27114 catalog_manager.cc:5495] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cf2f87cb790496882ab5c0299489e92 (127.19.34.65). New cstate: current_term: 1 leader_uuid: "1cf2f87cb790496882ab5c0299489e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cf2f87cb790496882ab5c0299489e92" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 } health_report { overall_health: UNKNOWN } } }
I20260619 14:52:14.915464 27627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.912959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:40815 (local address 127.19.34.65:34119)
0619 14:52:14.913079 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:14.913087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:14.913166 (+ 79us) server_negotiation.cc:407] Connection header received
0619 14:52:14.913254 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:14.913256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:14.913355 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:14.913413 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:52:14.913578 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.914016 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.914720 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.914899 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.914963 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.915133 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:52:14.915193 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:52:14.915239 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:14.915303 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 14:52:14.915356 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":62}
I20260619 14:52:14.915478 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.912824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34119 (local address 127.19.34.65:40815)
0619 14:52:14.913097 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:14.913110 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:14.913198 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:14.913428 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:14.913430 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:14.913436 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:52:14.913518 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:14.913521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.914090 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.914092 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:14.914620 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:14.914627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:14.914980 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:14.914982 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:14.915034 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:14.915054 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:52:14.915253 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:52:14.915256 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:52:14.915299 (+ 43us) client_negotiation.cc:240] Negotiation successful
0619 14:52:14.915369 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":2,"thread_start_us":64,"threads_started":1}
I20260619 14:52:14.920132 27414 raft_consensus.cc:1261] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 01932012b1bd4b8cb18f21da85bf1871 [term 1 FOLLOWER]: Refusing update from remote peer 1cf2f87cb790496882ab5c0299489e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:14.920217 27554 raft_consensus.cc:1261] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd [term 1 FOLLOWER]: Refusing update from remote peer 1cf2f87cb790496882ab5c0299489e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:14.920459 27636 consensus_queue.cc:1035] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01932012b1bd4b8cb18f21da85bf1871" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 43377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:14.920593 27684 consensus_queue.cc:1035] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9f8988d50b4448ea65bc250030db5bd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:14.945225 27114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:14.945374 27114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:14.947064 27114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98e17d1a0e7848939ad4ac94f9912f16: Sending DeleteTablet for 3 replicas of tablet 9cba96f7d7ae41c3bc190ac1f8758b6d
I20260619 14:52:14.947597 27394 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cba96f7d7ae41c3bc190ac1f8758b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:14 UTC) from {username='oryx', principal='oryx/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:33426
I20260619 14:52:14.947619 27534 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cba96f7d7ae41c3bc190ac1f8758b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:14 UTC) from {username='oryx', principal='oryx/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:59850
I20260619 14:52:14.947768 27652 tablet_replica.cc:330] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 01932012b1bd4b8cb18f21da85bf1871: stopping tablet replica
I20260619 14:52:14.947779 27254 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cba96f7d7ae41c3bc190ac1f8758b6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:14 UTC) from {username='oryx', principal='oryx/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:44702
I20260619 14:52:14.947880 27653 tablet_replica.cc:330] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd: stopping tablet replica
I20260619 14:52:14.947911 27652 raft_consensus.cc:2229] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 01932012b1bd4b8cb18f21da85bf1871 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:14.947943 27654 tablet_replica.cc:330] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92: stopping tablet replica
I20260619 14:52:14.947993 27653 raft_consensus.cc:2229] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:14.948017 27652 raft_consensus.cc:2258] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 01932012b1bd4b8cb18f21da85bf1871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:14.948043 27654 raft_consensus.cc:2229] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:52:14.948084 27653 raft_consensus.cc:2258] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:14.948199 27654 raft_consensus.cc:2258] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:14.948954 27654 ts_tablet_manager.cc:1905] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 1cf2f87cb790496882ab5c0299489e92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:14.949113 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 27189
I20260619 14:52:14.949119 27653 ts_tablet_manager.cc:1905] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:14.951354 27653 ts_tablet_manager.cc:1918] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:52:14.951421 27653 log.cc:1196] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/9cba96f7d7ae41c3bc190ac1f8758b6d
I20260619 14:52:14.951651 27653 ts_tablet_manager.cc:1939] T 9cba96f7d7ae41c3bc190ac1f8758b6d P b9f8988d50b4448ea65bc250030db5bd: Deleting consensus metadata
I20260619 14:52:14.952245 27101 catalog_manager.cc:4841] TS b9f8988d50b4448ea65bc250030db5bd (127.19.34.67:41781): tablet 9cba96f7d7ae41c3bc190ac1f8758b6d (table test-table [id=d2e2be58bd5d4eb48083e5835e6be86e]) successfully deleted
W20260619 14:52:14.956034 27104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.34.65:34119 (error 108)
I20260619 14:52:14.956107 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 27329
I20260619 14:52:14.956377 27652 ts_tablet_manager.cc:1905] T 9cba96f7d7ae41c3bc190ac1f8758b6d P 01932012b1bd4b8cb18f21da85bf1871: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:14.957865 27719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.957479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34119 (local address 127.0.0.1:44760)
0619 14:52:14.957711 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:14.957782 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.65:34119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":59,"threads_started":1}
W20260619 14:52:14.958102 27104 catalog_manager.cc:4584] TS 1cf2f87cb790496882ab5c0299489e92 (127.19.34.65:34119): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9cba96f7d7ae41c3bc190ac1f8758b6d: Network error: Client connection negotiation failed: client connection to 127.19.34.65:34119: connect: Connection refused (error 111)
I20260619 14:52:14.962276 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 27469
I20260619 14:52:14.962811 27719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:14.962310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:43377 (local address 127.0.0.1:33444)
0619 14:52:14.962431 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:14.962481 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.66:43377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 14:52:14.967490 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 27085
2026-06-19T14:52:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3229 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:52:15 dist-test-slave-3qdh krb5kdc[27722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:52:15 dist-test-slave-3qdh krb5kdc[27722](info): set up 2 sockets
Jun 19 14:52:15 dist-test-slave-3qdh krb5kdc[27722](info): commencing operation
krb5kdc: starting...
W20260619 14:52:15.940999 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:52:16 dist-test-slave-3qdh krb5kdc[27722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:52:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:52:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:52:16.048347 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:39041
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:39041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:16.144716 27735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:16.144922 27735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:16.145005 27735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:16.148777 27735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:16.148823 27735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:16.148842 27735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:16.148914 27735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:16.148947 27735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:52:16.153494 27735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:39041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:39041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:16.154340 27735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:16.155268 27735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:52:16.162767 27735 server_base.cc:1034] running on GCE node
W20260619 14:52:16.162782 27741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:16.162782 27740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:16.162920 27743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:16.163445 27735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:16.164410 27735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:16.165573 27735 hybrid_clock.cc:648] HybridClock initialized: now 1781880736165557 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T14:52:16Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1206 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:52:16 dist-test-slave-3qdh krb5kdc[27748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:52:16 dist-test-slave-3qdh krb5kdc[27748](info): set up 2 sockets
Jun 19 14:52:16 dist-test-slave-3qdh krb5kdc[27748](info): commencing operation
krb5kdc: starting...
W20260619 14:52:17.160910 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:52:17 dist-test-slave-3qdh krb5kdc[27748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:52:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:52:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:52:17.269217 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:44491
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:44491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:17.368634 27761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:17.368860 27761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:17.368899 27761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:17.372421 27761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:17.372468 27761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:17.372488 27761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:17.372540 27761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:17.372574 27761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:52:17.377557 27761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:44491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:44491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:17.378443 27761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:17.379276 27761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:17.386999 27766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:17.387079 27767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:17.387001 27769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:17.387233 27761 server_base.cc:1034] running on GCE node
I20260619 14:52:17.387717 27761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:17.388763 27761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:17.390110 27761 hybrid_clock.cc:648] HybridClock initialized: now 1781880737390084 us; error 43 us; skew 500 ppm
Jun 19 14:52:17 dist-test-slave-3qdh krb5kdc[27748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880737, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:17.392560 27761 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:52:17.393252 27761 webserver.cc:460] Webserver started at http://127.19.34.126:44895/ using document root <none> and password file <none>
I20260619 14:52:17.393673 27761 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:17.393721 27761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:17.393850 27761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:17.395624 27761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "8d2acca09f134935b773ab5f0e26629c"
format_stamp: "Formatted at 2026-06-19 14:52:17 on dist-test-slave-3qdh"
server_key: "0ada8b95d6aea694539b8668dcc9e9ec"
server_key_iv: "2d08ba9b38d32813e5480a0e1a6f7e26"
server_key_version: "encryptionkey@0"
I20260619 14:52:17.396107 27761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "8d2acca09f134935b773ab5f0e26629c"
format_stamp: "Formatted at 2026-06-19 14:52:17 on dist-test-slave-3qdh"
server_key: "0ada8b95d6aea694539b8668dcc9e9ec"
server_key_iv: "2d08ba9b38d32813e5480a0e1a6f7e26"
server_key_version: "encryptionkey@0"
I20260619 14:52:17.399276 27761 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:52:17.401209 27776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:17.402232 27761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:52:17.402360 27761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "8d2acca09f134935b773ab5f0e26629c"
format_stamp: "Formatted at 2026-06-19 14:52:17 on dist-test-slave-3qdh"
server_key: "0ada8b95d6aea694539b8668dcc9e9ec"
server_key_iv: "2d08ba9b38d32813e5480a0e1a6f7e26"
server_key_version: "encryptionkey@0"
I20260619 14:52:17.402454 27761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:17.424526 27761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:17.427074 27761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:17.427246 27761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:17.433655 27761 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:44491
I20260619 14:52:17.433760 27828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:44491 every 8 connection(s)
I20260619 14:52:17.434489 27761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:52:17.436966 27829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:17.441496 27829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c: Bootstrap starting.
I20260619 14:52:17.443624 27829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:17.443984 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 27761
I20260619 14:52:17.444111 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:52:17.444440 19593 external_mini_cluster.cc:1428] Setting key 20f0a1bffc848cbe79b1ac42f6e3c3c6
I20260619 14:52:17.444434 27829 log.cc:824] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:17.446437 27829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c: No bootstrap required, opened a new log
I20260619 14:52:17.450220 27829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2acca09f134935b773ab5f0e26629c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44491 } }
I20260619 14:52:17.450433 27829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:17.450480 27829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2acca09f134935b773ab5f0e26629c, State: Initialized, Role: FOLLOWER
I20260619 14:52:17.450870 27829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2acca09f134935b773ab5f0e26629c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44491 } }
I20260619 14:52:17.451005 27829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:52:17.451073 27829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:52:17.451169 27829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:17.452023 27829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2acca09f134935b773ab5f0e26629c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44491 } }
I20260619 14:52:17.452394 27829 leader_election.cc:302] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d2acca09f134935b773ab5f0e26629c; no voters:
I20260619 14:52:17.452721 27829 leader_election.cc:288] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:52:17.452785 27834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [term 1 FOLLOWER]: Leader election won for term 1
Jun 19 14:52:17 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:52:17.453022 27834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [term 1 LEADER]: Becoming Leader. State: Replica: 8d2acca09f134935b773ab5f0e26629c, State: Running, Role: LEADER
I20260619 14:52:17.453343 27834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2acca09f134935b773ab5f0e26629c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44491 } }
I20260619 14:52:17.453866 27829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:17.455235 27835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d2acca09f134935b773ab5f0e26629c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2acca09f134935b773ab5f0e26629c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44491 } } }
I20260619 14:52:17.455358 27835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:17.455288 27836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d2acca09f134935b773ab5f0e26629c. Latest consensus state: current_term: 1 leader_uuid: "8d2acca09f134935b773ab5f0e26629c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2acca09f134935b773ab5f0e26629c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44491 } } }
I20260619 14:52:17.455407 27836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:17.455685 27848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:52:17.458493 27833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:17.445463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47524 (local address 127.19.34.126:44491)
0619 14:52:17.445883 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:17.445901 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:17.445940 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:52:17.446713 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:17.446737 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:17.447110 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:17.447416 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:17.447992 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:17.449343 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:17.450138 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:17.450754 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:17.450893 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:17.453562 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:17.453568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:17.453580 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:17.453617 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:17.455544 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:17.456054 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:17.456062 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:17.456070 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:17.456212 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:17.456538 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:17.456542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:17.456543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:17.457024 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:17.457224 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:17.457768 (+ 544us) server_negotiation.cc:299] Negotiation successful
0619 14:52:17.457950 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":116,"threads_started":1}
I20260619 14:52:17.458838 27848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:52:17.463485 27848 catalog_manager.cc:1347] Generated new cluster ID: bc87aefa0200444a9cd790b698a5b362
I20260619 14:52:17.463542 27848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:52:17.473212 27848 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:52:17.474233 27848 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:52:17.483180 27848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c: Generated new TSK 0
I20260619 14:52:17.483777 27848 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.34.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
I20260619 14:52:17.509104 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44491
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:17.607409 27855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:17.607612 27855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:17.607649 27855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:17.611531 27855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:17.611579 27855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:17.611675 27855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:52:17.616715 27855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:17.617547 27855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:17.618422 27855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:17.626722 27860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:17.626753 27863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:17.626724 27861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:17.626943 27855 server_base.cc:1034] running on GCE node
I20260619 14:52:17.627502 27855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:17.627996 27855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:17.629143 27855 hybrid_clock.cc:648] HybridClock initialized: now 1781880737629121 us; error 33 us; skew 500 ppm
Jun 19 14:52:17 dist-test-slave-3qdh krb5kdc[27748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880737, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:17.631557 27855 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:52:17.632262 27855 webserver.cc:460] Webserver started at http://127.19.34.65:42979/ using document root <none> and password file <none>
I20260619 14:52:17.632781 27855 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:17.632838 27855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:17.632979 27855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:17.634749 27855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "dc69eaddaead44049931e4bbb0822558"
format_stamp: "Formatted at 2026-06-19 14:52:17 on dist-test-slave-3qdh"
server_key: "5621bd938a4dd3003603faef395b0d8d"
server_key_iv: "2160349bf73780fa2474859fcee3de12"
server_key_version: "encryptionkey@0"
I20260619 14:52:17.635193 27855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "dc69eaddaead44049931e4bbb0822558"
format_stamp: "Formatted at 2026-06-19 14:52:17 on dist-test-slave-3qdh"
server_key: "5621bd938a4dd3003603faef395b0d8d"
server_key_iv: "2160349bf73780fa2474859fcee3de12"
server_key_version: "encryptionkey@0"
I20260619 14:52:17.638523 27855 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260619 14:52:17.640579 27870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:17.641575 27855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:52:17.641665 27855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "dc69eaddaead44049931e4bbb0822558"
format_stamp: "Formatted at 2026-06-19 14:52:17 on dist-test-slave-3qdh"
server_key: "5621bd938a4dd3003603faef395b0d8d"
server_key_iv: "2160349bf73780fa2474859fcee3de12"
server_key_version: "encryptionkey@0"
I20260619 14:52:17.641731 27855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:17.655040 27855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:17.657624 27855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:17.657749 27855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:17.658179 27855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:17.658803 27855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:17.658842 27855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:17.658875 27855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:17.658888 27855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:17.666774 27855 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:45665
I20260619 14:52:17.666795 27983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:45665 every 8 connection(s)
I20260619 14:52:17.667487 27855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:52:17.673830 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 27855
I20260619 14:52:17.673949 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:52:17.674217 19593 external_mini_cluster.cc:1428] Setting key 7c0b97b9a067f92a1c29d0c5137127a7
Jun 19 14:52:17 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880737, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:52:17.679966 27833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:17.669033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:35737 (local address 127.19.34.126:44491)
0619 14:52:17.669153 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:17.669156 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:17.669754 (+ 598us) server_negotiation.cc:407] Connection header received
0619 14:52:17.670839 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:17.670842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:17.670913 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:17.671017 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:17.672041 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:17.672664 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:17.673735 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:17.674062 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:17.674170 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:17.676810 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:17.676816 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:17.676822 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:17.676854 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:17.678196 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:17.678638 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:17.678654 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:17.678668 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:17.678737 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:17.679037 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:17.679042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:17.679043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:17.679213 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:17.679311 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:17.679707 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 14:52:17.679818 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:52:17.680624 27986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:17.669259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.65:35737)
0619 14:52:17.669632 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:17.669667 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:17.670662 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:17.671163 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:17.671169 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:17.671497 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:17.671903 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:17.671915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:17.672772 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:17.672775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:17.673611 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:17.673619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:17.674204 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:17.674208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:17.674419 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:17.674941 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:17.674960 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:17.676655 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:17.678290 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:17.678294 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:17.678302 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:17.678542 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:17.678827 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:17.678830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:17.678832 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:17.678946 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:17.679308 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:17.679313 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:17.679503 (+ 190us) client_negotiation.cc:769] Sending connection context
0619 14:52:17.679661 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 14:52:17.679913 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":104,"threads_started":1}
I20260619 14:52:17.681861 27984 heartbeater.cc:347] Connected to a master server at 127.19.34.126:44491
I20260619 14:52:17.682087 27984 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:17.682451 27984 heartbeater.cc:511] Master 127.19.34.126:44491 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.34.66@KRBTEST.COM; defaulting to no policy
I20260619 14:52:17.683821 27793 ts_manager.cc:194] Registered new tserver with Master: dc69eaddaead44049931e4bbb0822558 (127.19.34.65:45665)
I20260619 14:52:17.684867 27793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:35737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
I20260619 14:52:17.699479 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44491
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:17.798128 27989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:17.798333 27989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:17.798401 27989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:17.802309 27989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:17.802354 27989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:17.802454 27989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:52:17.807406 27989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:17.808272 27989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:17.809212 27989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:17.817468 27997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:17.817519 27994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:17.817710 27989 server_base.cc:1034] running on GCE node
W20260619 14:52:17.817476 27995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:17.818061 27989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:17.818589 27989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:17.819774 27989 hybrid_clock.cc:648] HybridClock initialized: now 1781880737819758 us; error 39 us; skew 500 ppm
Jun 19 14:52:17 dist-test-slave-3qdh krb5kdc[27748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880737, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:17.822103 27989 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:52:17.822793 27989 webserver.cc:460] Webserver started at http://127.19.34.66:43573/ using document root <none> and password file <none>
I20260619 14:52:17.823275 27989 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:17.823354 27989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:17.823547 27989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:17.825452 27989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "39991fa428a94595acbb387c526a7b04"
format_stamp: "Formatted at 2026-06-19 14:52:17 on dist-test-slave-3qdh"
server_key: "19965fc7163beb8c3b9fb3352e1036c6"
server_key_iv: "d21274d5f63b9431ab0bde94d485cc09"
server_key_version: "encryptionkey@0"
I20260619 14:52:17.825965 27989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "39991fa428a94595acbb387c526a7b04"
format_stamp: "Formatted at 2026-06-19 14:52:17 on dist-test-slave-3qdh"
server_key: "19965fc7163beb8c3b9fb3352e1036c6"
server_key_iv: "d21274d5f63b9431ab0bde94d485cc09"
server_key_version: "encryptionkey@0"
I20260619 14:52:17.829295 27989 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 14:52:17.831460 28004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:17.832659 27989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:52:17.832777 27989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "39991fa428a94595acbb387c526a7b04"
format_stamp: "Formatted at 2026-06-19 14:52:17 on dist-test-slave-3qdh"
server_key: "19965fc7163beb8c3b9fb3352e1036c6"
server_key_iv: "d21274d5f63b9431ab0bde94d485cc09"
server_key_version: "encryptionkey@0"
I20260619 14:52:17.832896 27989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:17.849766 27989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:17.852031 27989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:17.852196 27989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:17.852690 27989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:17.853366 27989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:17.853425 27989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:17.853497 27989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:17.853546 27989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:17.861465 27989 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:36177
I20260619 14:52:17.861486 28117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:36177 every 8 connection(s)
I20260619 14:52:17.862180 27989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:52:17.864981 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 27989
I20260619 14:52:17.865144 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:52:17.865382 19593 external_mini_cluster.cc:1428] Setting key 33bc75ed3c11c1a611b5991f043a1cec
Jun 19 14:52:17 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880737, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.34.67@KRBTEST.COM; defaulting to no policy
I20260619 14:52:17.875114 27833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:17.863672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:50137 (local address 127.19.34.126:44491)
0619 14:52:17.863809 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:17.863813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:17.864614 (+ 801us) server_negotiation.cc:407] Connection header received
0619 14:52:17.865890 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:17.865894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:17.865947 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:17.866039 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:17.867260 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:17.868073 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:17.869087 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:17.869449 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:17.869557 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:17.871922 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:17.871925 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:17.871926 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:17.871946 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:17.873307 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:17.873837 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:17.873839 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:17.873840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:17.873901 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:17.874215 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:17.874217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:17.874218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:17.874405 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:17.874478 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:17.874928 (+ 450us) server_negotiation.cc:299] Negotiation successful
0619 14:52:17.875012 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:52:17.875789 28120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:17.864028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.66:50137)
0619 14:52:17.864474 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:17.864509 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:17.865635 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:17.866205 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:17.866213 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:17.866635 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:17.867109 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:17.867121 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:17.868181 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:17.868183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:17.868972 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:17.868980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:17.869568 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:17.869570 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:17.869679 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:17.870152 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:17.870165 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:17.871752 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:17.873425 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:17.873431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:17.873448 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:17.873748 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:17.873980 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:17.873982 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:17.873984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:17.874136 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:17.874505 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:17.874510 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:17.874747 (+ 237us) client_negotiation.cc:769] Sending connection context
0619 14:52:17.874886 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 14:52:17.875077 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":124,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.67@KRBTEST.COM" created.
I20260619 14:52:17.876861 28118 heartbeater.cc:347] Connected to a master server at 127.19.34.126:44491
I20260619 14:52:17.877091 28118 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:17.877499 28118 heartbeater.cc:511] Master 127.19.34.126:44491 requested a full tablet report, sending...
I20260619 14:52:17.878444 27793 ts_manager.cc:194] Registered new tserver with Master: 39991fa428a94595acbb387c526a7b04 (127.19.34.66:36177)
I20260619 14:52:17.878957 27793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:50137
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
I20260619 14:52:17.888955 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44491
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:17.986616 28123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:17.986827 28123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:17.986894 28123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:17.990655 28123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:17.990696 28123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:17.990782 28123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:52:17.995312 28123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:17.996133 28123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:17.997053 28123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:52:18.004750 28123 server_base.cc:1034] running on GCE node
W20260619 14:52:18.004714 28129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:18.004712 28128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:18.004719 28131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:18.005280 28123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:18.005859 28123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:18.007061 28123 hybrid_clock.cc:648] HybridClock initialized: now 1781880738007038 us; error 29 us; skew 500 ppm
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880738, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:18.009514 28123 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:52:18.010205 28123 webserver.cc:460] Webserver started at http://127.19.34.67:41103/ using document root <none> and password file <none>
I20260619 14:52:18.010674 28123 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:18.010761 28123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:18.010953 28123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:18.012678 28123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "349c9d1ceb0f43a99a9eba5bb326315e"
format_stamp: "Formatted at 2026-06-19 14:52:18 on dist-test-slave-3qdh"
server_key: "1c2d9038ef19e826c1f126845fdaf694"
server_key_iv: "250c21c6b56ffe04c5c0b78e697e0159"
server_key_version: "encryptionkey@0"
I20260619 14:52:18.013198 28123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "349c9d1ceb0f43a99a9eba5bb326315e"
format_stamp: "Formatted at 2026-06-19 14:52:18 on dist-test-slave-3qdh"
server_key: "1c2d9038ef19e826c1f126845fdaf694"
server_key_iv: "250c21c6b56ffe04c5c0b78e697e0159"
server_key_version: "encryptionkey@0"
I20260619 14:52:18.016263 28123 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:52:18.018335 28138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:18.019340 28123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:18.019456 28123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "349c9d1ceb0f43a99a9eba5bb326315e"
format_stamp: "Formatted at 2026-06-19 14:52:18 on dist-test-slave-3qdh"
server_key: "1c2d9038ef19e826c1f126845fdaf694"
server_key_iv: "250c21c6b56ffe04c5c0b78e697e0159"
server_key_version: "encryptionkey@0"
I20260619 14:52:18.019549 28123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:18.030709 28123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:18.033084 28123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:18.033257 28123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:18.033704 28123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:18.034361 28123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:18.034425 28123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:18.034478 28123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:18.034525 28123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:18.042074 28123 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:46687
I20260619 14:52:18.042129 28251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:46687 every 8 connection(s)
I20260619 14:52:18.042759 28123 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:52:18.043349 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 28123
I20260619 14:52:18.043501 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:52:18.043780 19593 external_mini_cluster.cc:1428] Setting key 3607ba12c533c20cebdb0cae75f0dcbe
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880738, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:52:18.054651 27833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:18.044365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:50509 (local address 127.19.34.126:44491)
0619 14:52:18.044475 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:18.044478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:18.045409 (+ 931us) server_negotiation.cc:407] Connection header received
0619 14:52:18.046614 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:18.046617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:18.046668 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:18.046743 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:18.047683 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.048505 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.049463 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.049672 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.049764 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:18.051797 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:18.051800 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:18.051803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:18.051824 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:18.052954 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:18.053371 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:18.053374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:18.053376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:18.053445 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:18.053733 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:18.053737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:18.053739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:18.053890 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:18.053979 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:18.054410 (+ 431us) server_negotiation.cc:299] Negotiation successful
0619 14:52:18.054510 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:52:18.055265 28254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:18.044810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.67:50509)
0619 14:52:18.045295 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:18.045327 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:18.046428 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:18.046892 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:18.046898 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:18.047196 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:18.047545 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:18.047554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.048604 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.048607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:18.049354 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:18.049362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.049805 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.049808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:18.049964 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:18.050420 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:18.050433 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:18.051600 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:18.053052 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:18.053057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:18.053068 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:18.053274 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:18.053522 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:18.053526 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:18.053528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:18.053649 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:18.053977 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:18.053982 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:18.054225 (+ 243us) client_negotiation.cc:769] Sending connection context
0619 14:52:18.054377 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 14:52:18.054627 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":112,"threads_started":1}
I20260619 14:52:18.056145 28252 heartbeater.cc:347] Connected to a master server at 127.19.34.126:44491
I20260619 14:52:18.056381 28252 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:18.056872 28252 heartbeater.cc:511] Master 127.19.34.126:44491 requested a full tablet report, sending...
I20260619 14:52:18.057775 27793 ts_manager.cc:194] Registered new tserver with Master: 349c9d1ceb0f43a99a9eba5bb326315e (127.19.34.67:46687)
I20260619 14:52:18.058256 27793 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:50509
I20260619 14:52:18.068816 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.34.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.34.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.34.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.65.keytab.
Entry for principal oryx/127.19.34.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.65.keytab.
WARNING: no policy specified for oryx/127.19.34.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.34.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.34.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.66.keytab.
Entry for principal oryx/127.19.34.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.66.keytab.
WARNING: no policy specified for oryx/127.19.34.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.34.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.34.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.67.keytab.
Entry for principal oryx/127.19.34.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.67.keytab.
I20260619 14:52:18.130683 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 27855
I20260619 14:52:18.135679 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 27989
I20260619 14:52:18.140579 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 28123
I20260619 14:52:18.144940 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 27761
I20260619 14:52:18.150280 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:44491
--webserver_interface=127.19.34.126
--webserver_port=44895
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:44491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:18.250511 28261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:18.250787 28261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:18.250855 28261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:18.254563 28261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:18.254649 28261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:18.254681 28261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:18.254719 28261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:18.254753 28261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:52:18.259330 28261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:44491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:44491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=44895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:18.260159 28261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:18.260993 28261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:18.268225 28269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:18.268245 28266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:18.268393 28267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:18.268433 28261 server_base.cc:1034] running on GCE node
I20260619 14:52:18.268932 28261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:18.269852 28261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:18.271052 28261 hybrid_clock.cc:648] HybridClock initialized: now 1781880738271033 us; error 30 us; skew 500 ppm
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880738, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:18.273638 28261 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:52:18.274359 28261 webserver.cc:460] Webserver started at http://127.19.34.126:44895/ using document root <none> and password file <none>
I20260619 14:52:18.274847 28261 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:18.274912 28261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:18.278200 28261 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 14:52:18.280018 28276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:18.281157 28261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:52:18.281270 28261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "8d2acca09f134935b773ab5f0e26629c"
format_stamp: "Formatted at 2026-06-19 14:52:17 on dist-test-slave-3qdh"
server_key: "0ada8b95d6aea694539b8668dcc9e9ec"
server_key_iv: "2d08ba9b38d32813e5480a0e1a6f7e26"
server_key_version: "encryptionkey@0"
I20260619 14:52:18.281714 28261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:18.299602 28261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:18.302562 28261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:18.302753 28261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:18.309370 28261 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:44491
I20260619 14:52:18.309396 28328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:44491 every 8 connection(s)
I20260619 14:52:18.310288 28261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:52:18.313982 28329 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:52:18.315426 28329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c: Bootstrap starting.
I20260619 14:52:18.315482 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 28261
I20260619 14:52:18.315915 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:45665
--local_ip_for_outbound_sockets=127.19.34.65
--tserver_master_addrs=127.19.34.126:44491
--webserver_port=42979
--webserver_interface=127.19.34.65
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.34.65
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:52:18.337218 28329 log.cc:824] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:18.345122 28329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:52:18.345526 28329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c: Bootstrap complete.
I20260619 14:52:18.351220 28329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2acca09f134935b773ab5f0e26629c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44491 } }
I20260619 14:52:18.351722 28329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d2acca09f134935b773ab5f0e26629c, State: Initialized, Role: FOLLOWER
I20260619 14:52:18.352216 28329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2acca09f134935b773ab5f0e26629c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44491 } }
I20260619 14:52:18.352372 28329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:52:18.352443 28329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:52:18.352535 28329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:52:18.354723 28329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2acca09f134935b773ab5f0e26629c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44491 } }
I20260619 14:52:18.355046 28329 leader_election.cc:302] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d2acca09f134935b773ab5f0e26629c; no voters:
I20260619 14:52:18.355353 28329 leader_election.cc:288] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:52:18.355466 28333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:52:18.355712 28333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [term 2 LEADER]: Becoming Leader. State: Replica: 8d2acca09f134935b773ab5f0e26629c, State: Running, Role: LEADER
I20260619 14:52:18.356045 28333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2acca09f134935b773ab5f0e26629c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44491 } }
I20260619 14:52:18.356602 28329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:18.357491 28335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d2acca09f134935b773ab5f0e26629c. Latest consensus state: current_term: 2 leader_uuid: "8d2acca09f134935b773ab5f0e26629c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2acca09f134935b773ab5f0e26629c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44491 } } }
I20260619 14:52:18.357470 28334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8d2acca09f134935b773ab5f0e26629c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d2acca09f134935b773ab5f0e26629c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44491 } } }
I20260619 14:52:18.357597 28334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:18.357605 28335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:18.357939 28342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:52:18.360795 28342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:52:18.361558 28342 catalog_manager.cc:1259] Loaded cluster ID: bc87aefa0200444a9cd790b698a5b362
I20260619 14:52:18.361629 28342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:52:18.363178 28342 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:52:18.363705 28342 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8d2acca09f134935b773ab5f0e26629c: Loaded TSK: 0
I20260619 14:52:18.364635 28342 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:52:18.441008 28331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:18.441227 28331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:18.441295 28331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:18.444890 28331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:18.444964 28331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:18.445052 28331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:52:18.449846 28331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.65.keytab
--principal=oryx/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:45665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=42979
--tserver_master_addrs=127.19.34.126:44491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:18.450699 28331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:18.451690 28331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:18.460434 28358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:18.460444 28356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:18.460628 28331 server_base.cc:1034] running on GCE node
W20260619 14:52:18.460413 28355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:18.461117 28331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:18.461805 28331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:18.462994 28331 hybrid_clock.cc:648] HybridClock initialized: now 1781880738462971 us; error 44 us; skew 500 ppm
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880738, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:18.465639 28331 init.cc:377] Logged in from keytab as oryx/127.19.34.65@KRBTEST.COM (short username oryx)
I20260619 14:52:18.466495 28331 webserver.cc:460] Webserver started at http://127.19.34.65:42979/ using document root <none> and password file <none>
I20260619 14:52:18.467003 28331 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:18.467067 28331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:18.470512 28331 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:52:18.472546 28365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:18.473742 28331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:18.473876 28331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "dc69eaddaead44049931e4bbb0822558"
format_stamp: "Formatted at 2026-06-19 14:52:17 on dist-test-slave-3qdh"
server_key: "5621bd938a4dd3003603faef395b0d8d"
server_key_iv: "2160349bf73780fa2474859fcee3de12"
server_key_version: "encryptionkey@0"
I20260619 14:52:18.474354 28331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:18.493242 28331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:18.496119 28331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:18.496346 28331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:18.496919 28331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:18.497663 28331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:18.497726 28331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:18.497786 28331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:18.497833 28331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:18.508059 28331 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:45665
I20260619 14:52:18.508116 28478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:45665 every 8 connection(s)
I20260619 14:52:18.509073 28331 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:52:18.512444 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 28331
I20260619 14:52:18.512822 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:36177
--local_ip_for_outbound_sockets=127.19.34.66
--tserver_master_addrs=127.19.34.126:44491
--webserver_port=43573
--webserver_interface=127.19.34.66
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.34.66
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:52:18.522375 28481 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:18.510962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:57475 (local address 127.19.34.126:44491)
0619 14:52:18.511329 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:18.511347 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:18.511652 (+ 305us) server_negotiation.cc:407] Connection header received
0619 14:52:18.513107 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:18.513117 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:18.513348 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:18.513629 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:18.514594 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.515493 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.517348 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.517982 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.518107 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:18.521722 (+ 3615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.65:57475: BlockingRecv error: recv got EOF from 127.19.34.65:57475 (error 108)
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":118,"threads_started":1}
I20260619 14:52:18.522382 28482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:18.511129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.65:57475)
0619 14:52:18.511507 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:18.511542 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:18.512759 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:18.513697 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:18.513704 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:18.514058 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:18.514435 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:18.514446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.515629 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.515635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:18.517206 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:18.517218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.518143 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.518151 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:18.518361 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:18.518942 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:18.518967 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:18.521549 (+ 2582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":101,"threads_started":1}
W20260619 14:52:18.522934 28479 heartbeater.cc:644] Failed to heartbeat to 127.19.34.126:44491 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.34.126:44491: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:18.527940 28482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:18.523317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.65:40083)
0619 14:52:18.523409 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:18.523423 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:18.523485 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:18.523716 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:18.523719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:18.523860 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:18.523932 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:18.523936 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.524660 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.524666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:18.525191 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:18.525199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.525623 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.525626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:18.525708 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:18.526166 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:18.526177 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:18.527831 (+ 1654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 14:52:18.528055 28481 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:18.523426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:40083 (local address 127.19.34.126:44491)
0619 14:52:18.523539 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:18.523545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:18.523557 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:52:18.523589 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:18.523591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:18.523628 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:18.523705 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:18.524032 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.524560 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.525307 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.525522 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.525628 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:18.527938 (+ 2310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.65:40083: BlockingRecv error: recv got EOF from 127.19.34.65:40083 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:18.532986 28484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:18.528359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.65:48841)
0619 14:52:18.528658 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:18.528669 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:18.528764 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:18.528975 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:18.528978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:18.529155 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:18.529228 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:18.529233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.529844 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.529848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:18.530351 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:18.530358 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.530755 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.530758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:18.530896 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:18.531283 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:18.531293 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:18.532833 (+ 1540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":95,"threads_started":1}
W20260619 14:52:18.533027 28481 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:18.528485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:48841 (local address 127.19.34.126:44491)
0619 14:52:18.528589 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:18.528594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:18.528732 (+ 138us) server_negotiation.cc:407] Connection header received
0619 14:52:18.528850 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:18.528853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:18.528892 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:18.528962 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:18.529327 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.529750 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.530457 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.530649 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.530745 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:18.532930 (+ 2185us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.65:48841: BlockingRecv error: recv got EOF from 127.19.34.65:48841 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260619 14:52:18.533254 28479 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:52:18.615597 28483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:18.615793 28483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:18.615851 28483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:18.619755 28483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:18.619798 28483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:18.619880 28483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:52:18.624447 28483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.66.keytab
--principal=oryx/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:36177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=43573
--tserver_master_addrs=127.19.34.126:44491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:18.625191 28483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:18.626072 28483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:18.634428 28489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:18.634516 28483 server_base.cc:1034] running on GCE node
W20260619 14:52:18.634433 28490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:18.634439 28492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:18.635069 28483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:18.635560 28483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:18.636701 28483 hybrid_clock.cc:648] HybridClock initialized: now 1781880738636676 us; error 36 us; skew 500 ppm
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880738, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:18.639045 28483 init.cc:377] Logged in from keytab as oryx/127.19.34.66@KRBTEST.COM (short username oryx)
I20260619 14:52:18.639745 28483 webserver.cc:460] Webserver started at http://127.19.34.66:43573/ using document root <none> and password file <none>
I20260619 14:52:18.640187 28483 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:18.640265 28483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:18.643198 28483 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:18.644882 28499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:18.645831 28483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:18.645929 28483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "39991fa428a94595acbb387c526a7b04"
format_stamp: "Formatted at 2026-06-19 14:52:17 on dist-test-slave-3qdh"
server_key: "19965fc7163beb8c3b9fb3352e1036c6"
server_key_iv: "d21274d5f63b9431ab0bde94d485cc09"
server_key_version: "encryptionkey@0"
I20260619 14:52:18.646392 28483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:18.662065 28483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:18.664412 28483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:18.664582 28483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:18.665089 28483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:18.665769 28483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:18.665813 28483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:18.665890 28483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:18.665930 28483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:18.673792 28483 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:36177
I20260619 14:52:18.673866 28612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:36177 every 8 connection(s)
I20260619 14:52:18.674605 28483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:52:18.679354 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 28483
I20260619 14:52:18.679726 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:46687
--local_ip_for_outbound_sockets=127.19.34.67
--tserver_master_addrs=127.19.34.126:44491
--webserver_port=41103
--webserver_interface=127.19.34.67
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:46409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.34.67
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:52:18.685087 28481 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:18.676120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:49181 (local address 127.19.34.126:44491)
0619 14:52:18.676270 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:18.676275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:18.677018 (+ 743us) server_negotiation.cc:407] Connection header received
0619 14:52:18.678318 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:18.678321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:18.678366 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:18.678434 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:18.679455 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.680020 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.681260 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.681447 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.681519 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:18.684969 (+ 3450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.66:49181: BlockingRecv error: recv got EOF from 127.19.34.66:49181 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:52:18.685618 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:18.676480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.66:49181)
0619 14:52:18.676893 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:18.676932 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:18.678158 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:18.678610 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:18.678616 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:18.678896 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:18.679299 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:18.679311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.680145 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.680149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:18.681140 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:18.681154 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.681580 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.681587 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:18.681781 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:18.682354 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:18.682376 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:18.684851 (+ 2475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":103,"threads_started":1}
W20260619 14:52:18.686017 28613 heartbeater.cc:644] Failed to heartbeat to 127.19.34.126:44491 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.34.126:44491: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:18.691207 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:18.686450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.66:54613)
0619 14:52:18.686563 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:18.686577 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:18.686648 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:18.686900 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:18.686902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:18.687060 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:18.687160 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:18.687165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.687827 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.687832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:18.688430 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:18.688438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.688850 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.688853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:18.688937 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:18.689379 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:18.689399 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:18.691084 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 14:52:18.691612 28481 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:18.686506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:54613 (local address 127.19.34.126:44491)
0619 14:52:18.686626 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:18.686629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:18.686641 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:52:18.686756 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:18.686759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:18.686804 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:18.686885 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:18.687260 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.687696 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.688537 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.688743 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.688850 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:18.691532 (+ 2682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.66:54613: BlockingRecv error: recv got EOF from 127.19.34.66:54613 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:18.696712 28615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:18.691690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.66:54083)
0619 14:52:18.691825 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:18.691839 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:18.691905 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:18.692195 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:18.692198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:18.692382 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:18.692470 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:18.692477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.693117 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.693121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:18.693950 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:18.693959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.694367 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.694371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:18.694457 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:18.694836 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:18.694851 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:18.696593 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":81}
W20260619 14:52:18.696830 28617 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:18.691753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:54083 (local address 127.19.34.126:44491)
0619 14:52:18.691998 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:18.692003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:18.692015 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:52:18.692048 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:18.692050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:18.692085 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:18.692154 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:18.692584 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.693021 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.694058 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.694266 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.694338 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:18.696703 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.66:54083: BlockingRecv error: recv got EOF from 127.19.34.66:54083 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":116,"threads_started":1}
W20260619 14:52:18.697139 28613 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:52:18.785441 28616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:18.785712 28616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:18.785751 28616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:18.789492 28616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:18.789534 28616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:18.789585 28616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:52:18.794052 28616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:46409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/krb5kdc/oryx_127.19.34.67.keytab
--principal=oryx/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:46687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=41103
--tserver_master_addrs=127.19.34.126:44491
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:18.794829 28616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:18.795662 28616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:18.803993 28623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:18.804031 28625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:18.804021 28622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:18.804235 28616 server_base.cc:1034] running on GCE node
I20260619 14:52:18.804608 28616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:18.805126 28616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:18.806274 28616 hybrid_clock.cc:648] HybridClock initialized: now 1781880738806250 us; error 30 us; skew 500 ppm
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880738, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:18.808962 28616 init.cc:377] Logged in from keytab as oryx/127.19.34.67@KRBTEST.COM (short username oryx)
I20260619 14:52:18.809684 28616 webserver.cc:460] Webserver started at http://127.19.34.67:41103/ using document root <none> and password file <none>
I20260619 14:52:18.810233 28616 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:18.810329 28616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:18.815336 28616 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:52:18.817414 28632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:18.818542 28616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:52:18.818670 28616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "349c9d1ceb0f43a99a9eba5bb326315e"
format_stamp: "Formatted at 2026-06-19 14:52:18 on dist-test-slave-3qdh"
server_key: "1c2d9038ef19e826c1f126845fdaf694"
server_key_iv: "250c21c6b56ffe04c5c0b78e697e0159"
server_key_version: "encryptionkey@0"
I20260619 14:52:18.819141 28616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:18.841370 28616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:18.843674 28616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:18.843850 28616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:18.844451 28616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:18.845149 28616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:18.845243 28616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:18.845315 28616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:18.845366 28616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:18.853596 28616 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:46687
I20260619 14:52:18.853655 28745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:46687 every 8 connection(s)
I20260619 14:52:18.854514 28616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:52:18.855759 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 28616
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:52:18.865324 28617 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:18.856184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:60085 (local address 127.19.34.126:44491)
0619 14:52:18.856331 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:18.856335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:18.857327 (+ 992us) server_negotiation.cc:407] Connection header received
0619 14:52:18.858944 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:18.858947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:18.858992 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:18.859062 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:18.860204 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.860801 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.861774 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.861975 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.862093 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:18.865171 (+ 3078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.67:60085: BlockingRecv error: recv got EOF from 127.19.34.67:60085 (error 108)
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":46}
I20260619 14:52:18.865864 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:18.856631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.67:60085)
0619 14:52:18.857176 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:18.857220 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:18.858737 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:18.859200 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:18.859205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:18.859595 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:18.860086 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:18.860100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.860906 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.860908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:18.861656 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:18.861664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.862106 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.862109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:18.862279 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:18.862729 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:18.862742 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:18.865047 (+ 2305us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":142,"threads_started":1}
I20260619 14:52:18.866192 28481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:18.857091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.19.34.126:44491)
0619 14:52:18.857203 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:18.857207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:18.857219 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:52:18.857259 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:18.857262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:18.857317 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:18.857401 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:18.858401 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.859166 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.860204 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.860531 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.860623 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:18.861890 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:18.861895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:18.861915 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:18.861939 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:18.863724 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:18.864250 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:18.864258 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:18.864263 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:18.864417 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:18.864672 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:18.864675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:18.864678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:18.865075 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:18.865248 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:18.865793 (+ 545us) server_negotiation.cc:299] Negotiation successful
0619 14:52:18.866052 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":53}
W20260619 14:52:18.866317 28746 heartbeater.cc:644] Failed to heartbeat to 127.19.34.126:44491 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.34.126:44491: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:18.871462 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:18.866683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.67:55103)
0619 14:52:18.866856 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:18.866869 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:18.866951 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:18.867216 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:18.867218 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:18.867323 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:18.867396 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:18.867400 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.868335 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.868342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:18.869048 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:18.869057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.869499 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.869502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:18.869586 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:18.869938 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:18.869948 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:18.871393 (+ 1445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113}
W20260619 14:52:18.871630 28617 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:18.866741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:55103 (local address 127.19.34.126:44491)
0619 14:52:18.866855 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:18.866858 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:18.866914 (+ 56us) server_negotiation.cc:407] Connection header received
0619 14:52:18.867048 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:18.867051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:18.867114 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:18.867185 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:18.867489 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.868208 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.869161 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.869394 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.869498 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:18.871515 (+ 2017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.67:55103: BlockingRecv error: recv got EOF from 127.19.34.67:55103 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:18 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:18.876180 28749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:18.871873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.67:36721)
0619 14:52:18.871999 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:18.872013 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:18.872080 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:18.872342 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:18.872345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:18.872482 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:18.872538 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:18.872542 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.873190 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.873195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:18.873683 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:18.873690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.874114 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.874117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:18.874192 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:18.874514 (+ 322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:18.874524 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:18.876056 (+ 1532us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 14:52:18.876264 28617 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:18.871953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:36721 (local address 127.19.34.126:44491)
0619 14:52:18.872078 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:18.872081 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:18.872093 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:52:18.872165 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:18.872169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:18.872200 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:18.872305 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:18.872657 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.873090 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.873792 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:18.874006 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:18.874108 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:18.876154 (+ 2046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.67:36721: BlockingRecv error: recv got EOF from 127.19.34.67:36721 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 14:52:18.876374 28746 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 14:52:19 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:19 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:19.539512 28750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:19.533877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.65:44391)
0619 14:52:19.534188 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:19.534206 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:19.534347 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:19.534640 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:19.534645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:19.534871 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:19.534973 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:19.534980 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:19.535823 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:19.535826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:19.536418 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:19.536426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:19.536862 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:19.536865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:19.536934 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:19.537499 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:19.537510 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:19.539345 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":86,"threads_started":1}
W20260619 14:52:19.539580 28751 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:19.533954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:44391 (local address 127.19.34.126:44491)
0619 14:52:19.534208 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:19.534213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:19.534289 (+ 76us) server_negotiation.cc:407] Connection header received
0619 14:52:19.534443 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:19.534448 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:19.534500 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:19.534592 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:19.535123 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:19.535702 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:19.536537 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:19.536761 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:19.536845 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:19.539461 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.65:44391: BlockingRecv error: recv got EOF from 127.19.34.65:44391 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":86,"threads_started":1}
Jun 19 14:52:19 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:19 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:19.702862 28752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:19.697706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.66:33883)
0619 14:52:19.697983 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:19.698003 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:19.698151 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:19.698418 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:19.698421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:19.698610 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:19.698714 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:19.698719 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:19.699423 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:19.699428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:19.700008 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:19.700016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:19.700528 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:19.700531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:19.700626 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:19.701127 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:19.701146 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:19.702764 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":88,"threads_started":1}
W20260619 14:52:19.703045 28751 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:19.697760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:33883 (local address 127.19.34.126:44491)
0619 14:52:19.697934 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:19.697938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:19.698073 (+ 135us) server_negotiation.cc:407] Connection header received
0619 14:52:19.698249 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:19.698252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:19.698302 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:19.698385 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:19.698822 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:19.699301 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:19.700142 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:19.700422 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:19.700524 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:19.702884 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.66:33883: BlockingRecv error: recv got EOF from 127.19.34.66:33883 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 19 14:52:19 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:19 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:19.881765 28753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:19.876903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.67:55925)
0619 14:52:19.877170 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:19.877182 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:19.877274 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:19.877496 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:19.877498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:19.877675 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:19.877815 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:19.877821 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:19.878526 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:19.878528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:19.879077 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:19.879084 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:19.879476 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:19.879479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:19.879535 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:19.879982 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:19.879992 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:19.881657 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":91,"threads_started":1}
W20260619 14:52:19.881887 28751 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:19.876954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:55925 (local address 127.19.34.126:44491)
0619 14:52:19.877067 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:19.877070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:19.877277 (+ 207us) server_negotiation.cc:407] Connection header received
0619 14:52:19.877347 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:19.877349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:19.877395 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:19.877464 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:19.877906 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:19.878433 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:19.879185 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:19.879389 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:19.879459 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:19.881772 (+ 2313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.67:55925: BlockingRecv error: recv got EOF from 127.19.34.67:55925 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 14:52:20 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:20 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:20.550607 28754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:20.540278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.65:37129)
0619 14:52:20.540544 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:20.540568 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:20.540667 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:20.540885 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:20.540887 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:20.541091 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:20.541188 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:20.541194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:20.541876 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:20.541879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:20.542402 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:20.542409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:20.542816 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:20.542818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:20.542875 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:20.548613 (+ 5738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:20.548626 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:20.550454 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":97,"threads_started":1}
W20260619 14:52:20.550706 28755 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:20.540367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:37129 (local address 127.19.34.126:44491)
0619 14:52:20.540571 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:20.540580 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:20.540622 (+ 42us) server_negotiation.cc:407] Connection header received
0619 14:52:20.540730 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:20.540731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:20.540783 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:20.540848 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:20.541286 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:20.541768 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:20.542529 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:20.542727 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:20.542794 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:20.550569 (+ 7775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.65:37129: BlockingRecv error: recv got EOF from 127.19.34.65:37129 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":51,"threads_started":1}
Jun 19 14:52:20 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:20 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:20.708549 28756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:20.703529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.66:50361)
0619 14:52:20.703779 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:20.703792 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:20.703878 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:20.704069 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:20.704071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:20.704257 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:20.704410 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:20.704417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:20.705040 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:20.705042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:20.705539 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:20.705545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:20.706037 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:20.706041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:20.706130 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:20.706710 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:20.706724 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:20.708447 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":104,"threads_started":1}
W20260619 14:52:20.708691 28755 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:20.703658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:50361 (local address 127.19.34.126:44491)
0619 14:52:20.703801 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:20.703804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:20.703853 (+ 49us) server_negotiation.cc:407] Connection header received
0619 14:52:20.703957 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:20.703959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:20.703996 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:20.704059 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:20.704501 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:20.704959 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:20.705647 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:20.705917 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:20.706040 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:20.708578 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.66:50361: BlockingRecv error: recv got EOF from 127.19.34.66:50361 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 14:52:20 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:20 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:20.887666 28757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:20.882578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.67:41121)
0619 14:52:20.882868 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:20.882881 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:20.883004 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:20.883246 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:20.883248 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:20.883420 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:20.883511 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:20.883516 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:20.884157 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:20.884162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:20.884713 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:20.884721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:20.885103 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:20.885107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:20.885200 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:20.885696 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:20.885715 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:20.887539 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":107,"threads_started":1}
W20260619 14:52:20.887750 28755 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:20.882715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:41121 (local address 127.19.34.126:44491)
0619 14:52:20.882839 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:20.882842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:20.882952 (+ 110us) server_negotiation.cc:407] Connection header received
0619 14:52:20.883095 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:20.883102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:20.883151 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:20.883225 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:20.883616 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:20.884063 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:20.884822 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:20.885011 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:20.885101 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:20.887646 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.67:41121: BlockingRecv error: recv got EOF from 127.19.34.67:41121 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 14:52:21 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:21 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:21.555976 28758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:21.551325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.65:55621)
0619 14:52:21.551551 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:21.551563 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:21.551638 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:21.551853 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:21.551855 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:21.552029 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:21.552108 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:21.552113 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:21.552775 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:21.552777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:21.553327 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:21.553333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:21.553677 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:21.553679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:21.553741 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:21.554150 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:21.554178 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:21.555870 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":74,"threads_started":1}
W20260619 14:52:21.556190 28759 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:21.551379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:55621 (local address 127.19.34.126:44491)
0619 14:52:21.551616 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:21.551619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:21.551630 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:52:21.551726 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:21.551728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:21.551764 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:21.551828 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:21.552199 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:21.552696 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:21.553410 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:21.553597 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:21.553660 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:21.555980 (+ 2320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.65:55621: BlockingRecv error: recv got EOF from 127.19.34.65:55621 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":78,"threads_started":1}
Jun 19 14:52:21 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:21 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:21.714017 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:21.709414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.66:49689)
0619 14:52:21.709666 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:21.709680 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:21.709770 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:21.709989 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:21.709991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:21.710176 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:21.710264 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:21.710269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:21.710904 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:21.710906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:21.711402 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:21.711408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:21.711779 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:21.711781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:21.711832 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:21.712244 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:21.712254 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:21.713897 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":79,"threads_started":1}
W20260619 14:52:21.714098 28759 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:21.709499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:49689 (local address 127.19.34.126:44491)
0619 14:52:21.709673 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:21.709678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:21.709744 (+ 66us) server_negotiation.cc:407] Connection header received
0619 14:52:21.709845 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:21.709847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:21.709889 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:21.709954 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:21.710356 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:21.710827 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:21.711505 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:21.711698 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:21.711765 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:21.714016 (+ 2251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.66:49689: BlockingRecv error: recv got EOF from 127.19.34.66:49689 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 19 14:52:21 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:21 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:21.892669 28761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:21.888271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.67:36717)
0619 14:52:21.888569 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:21.888593 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:21.888683 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:21.888883 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:21.888885 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:21.889037 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:21.889124 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:21.889129 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:21.889796 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:21.889798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:21.890266 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:21.890271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:21.890609 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:21.890611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:21.890675 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:21.891080 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:21.891089 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:21.892566 (+ 1477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":89,"threads_started":1}
W20260619 14:52:21.892783 28759 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:21.888361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:36717 (local address 127.19.34.126:44491)
0619 14:52:21.888499 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:21.888502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:21.888657 (+ 155us) server_negotiation.cc:407] Connection header received
0619 14:52:21.888743 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:21.888744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:21.888785 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:21.888846 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:21.889255 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:21.889713 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:21.890349 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:21.890530 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:21.890604 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:21.892681 (+ 2077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.67:36717: BlockingRecv error: recv got EOF from 127.19.34.67:36717 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 19 14:52:22 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:22 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:22.562290 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:22.556746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.65:52131)
0619 14:52:22.557045 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:22.557061 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:22.557220 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:22.557524 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:22.557526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:22.557702 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:22.557783 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:22.557788 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:22.558519 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:22.558523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:22.559130 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:22.559139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:22.559569 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:22.559572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:22.559650 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:22.560209 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:22.560220 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:22.562143 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":89,"threads_started":1}
W20260619 14:52:22.562342 28763 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:22.556828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:52131 (local address 127.19.34.126:44491)
0619 14:52:22.557120 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:22.557125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:22.557157 (+ 32us) server_negotiation.cc:407] Connection header received
0619 14:52:22.557350 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:22.557353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:22.557406 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:22.557484 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:22.557887 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:22.558421 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:22.559253 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:22.559447 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:22.559557 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:22.562243 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.65:52131: BlockingRecv error: recv got EOF from 127.19.34.65:52131 (error 108)
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":113,"threads_started":1}
Jun 19 14:52:22 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:22 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:22.719873 28764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:22.714833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.66:53251)
0619 14:52:22.715058 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:22.715069 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:22.715156 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:22.715348 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:22.715350 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:22.715495 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:22.715579 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:22.715584 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:22.716236 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:22.716238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:22.716954 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:22.716966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:22.717511 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:22.717517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:22.717596 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:22.718028 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:22.718038 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:22.719712 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
W20260619 14:52:22.719928 28763 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:22.714876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:53251 (local address 127.19.34.126:44491)
0619 14:52:22.715002 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:22.715006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:22.715128 (+ 122us) server_negotiation.cc:407] Connection header received
0619 14:52:22.715240 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:22.715242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:22.715279 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:22.715337 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:22.715659 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:22.716126 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:22.717058 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:22.717399 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:22.717480 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:22.719816 (+ 2336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.66:53251: BlockingRecv error: recv got EOF from 127.19.34.66:53251 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 19 14:52:22 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:22 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:22.898660 28765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:22.893412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.67:40109)
0619 14:52:22.893650 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:22.893664 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:22.893753 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:22.894023 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:22.894025 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:22.894188 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:22.894274 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:22.894278 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:22.894933 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:22.894935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:22.895506 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:22.895514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:22.895945 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:22.895948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:22.896007 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:22.896515 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:22.896526 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:22.898552 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
W20260619 14:52:22.898888 28763 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:22.893483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:40109 (local address 127.19.34.126:44491)
0619 14:52:22.893649 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:22.893653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:22.893709 (+ 56us) server_negotiation.cc:407] Connection header received
0619 14:52:22.893866 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:22.893868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:22.893917 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:22.894000 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:22.894368 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:22.894846 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:22.895615 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:22.895840 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:22.895934 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:22.898679 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.67:40109: BlockingRecv error: recv got EOF from 127.19.34.67:40109 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 14:52:23 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:23 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:23.567814 28766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:23.562934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.65:58435)
0619 14:52:23.563169 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:23.563181 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:23.563256 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:23.563482 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:23.563484 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:23.563633 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:23.563715 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:23.563720 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:23.564377 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:23.564379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:23.564843 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:23.564850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:23.565207 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:23.565209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:23.565261 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:23.565738 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:23.565748 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:23.567706 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
W20260619 14:52:23.567992 28767 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:23.563083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:58435 (local address 127.19.34.126:44491)
0619 14:52:23.563308 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:23.563311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:23.563321 (+ 10us) server_negotiation.cc:407] Connection header received
0619 14:52:23.563350 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:23.563351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:23.563385 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:23.563451 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:23.563802 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:23.564256 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:23.564928 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:23.565117 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:23.565183 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:23.567836 (+ 2653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.65:58435: BlockingRecv error: recv got EOF from 127.19.34.65:58435 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
Jun 19 14:52:23 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:23 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:23.725679 28768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:23.720682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.66:43395)
0619 14:52:23.720922 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:23.720933 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:23.721020 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:23.721227 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:23.721229 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:23.721383 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:23.721467 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:23.721471 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:23.722162 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:23.722165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:23.722728 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:23.722736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:23.723121 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:23.723125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:23.723221 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:23.723765 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:23.723782 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:23.725572 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":92,"threads_started":1}
W20260619 14:52:23.725797 28767 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:23.720768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:43395 (local address 127.19.34.126:44491)
0619 14:52:23.720889 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:23.720892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:23.720997 (+ 105us) server_negotiation.cc:407] Connection header received
0619 14:52:23.721095 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:23.721097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:23.721133 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:23.721194 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:23.721564 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:23.722037 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:23.722816 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:23.722997 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:23.723097 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:23.725693 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.66:43395: BlockingRecv error: recv got EOF from 127.19.34.66:43395 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 19 14:52:23 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:23 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:23.904500 28769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:23.899491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.67:35989)
0619 14:52:23.899770 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:23.899782 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:23.899894 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:23.900106 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:23.900109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:23.900290 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:23.900426 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:23.900431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:23.901062 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:23.901067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:23.901660 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:23.901668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:23.902058 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:23.902061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:23.902134 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:23.902598 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:23.902610 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:23.904402 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":82,"threads_started":1}
W20260619 14:52:23.904686 28767 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:23.899566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:35989 (local address 127.19.34.126:44491)
0619 14:52:23.899715 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:23.899719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:23.899849 (+ 130us) server_negotiation.cc:407] Connection header received
0619 14:52:23.899968 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:23.899971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:23.900017 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:23.900088 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:23.900519 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:23.900965 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:23.901763 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:23.901960 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:23.902054 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:23.904525 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.67:35989: BlockingRecv error: recv got EOF from 127.19.34.67:35989 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 14:52:24 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:24 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:24.573611 28770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:24.568617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.65:33129)
0619 14:52:24.568992 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:24.569005 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:24.569131 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:24.569467 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:24.569472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:24.569694 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:24.569788 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:24.569793 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:24.570451 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:24.570453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:24.570946 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:24.570952 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:24.571323 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:24.571325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:24.571377 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:24.571878 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:24.571890 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:24.573515 (+ 1625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":83,"threads_started":1}
W20260619 14:52:24.573755 28771 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:24.568686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:33129 (local address 127.19.34.126:44491)
0619 14:52:24.568922 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:24.568926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:24.569080 (+ 154us) server_negotiation.cc:407] Connection header received
0619 14:52:24.569236 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:24.569241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:24.569295 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:24.569396 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:24.569882 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:24.570371 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:24.571034 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:24.571239 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:24.571307 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:24.573636 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.65:33129: BlockingRecv error: recv got EOF from 127.19.34.65:33129 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":66,"threads_started":1}
Jun 19 14:52:24 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:24 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:24.731936 28772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:24.726566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.66:46001)
0619 14:52:24.727026 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:24.727038 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:24.727164 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:24.727426 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:24.727428 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:24.727642 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:24.727725 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:24.727730 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:24.728537 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:24.728543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:24.729081 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:24.729088 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:24.729547 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:24.729550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:24.729645 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:24.730070 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:24.730080 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:24.731779 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":88,"threads_started":1}
W20260619 14:52:24.731978 28771 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:24.726617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:46001 (local address 127.19.34.126:44491)
0619 14:52:24.726784 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:24.726788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:24.727120 (+ 332us) server_negotiation.cc:407] Connection header received
0619 14:52:24.727270 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:24.727273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:24.727326 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:24.727407 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:24.727847 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:24.728439 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:24.729202 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:24.729438 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:24.729537 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:24.731882 (+ 2345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.66:46001: BlockingRecv error: recv got EOF from 127.19.34.66:46001 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 14:52:24 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:24 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:24.910704 28773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:24.905225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.67:33089)
0619 14:52:24.905477 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:24.905490 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:24.905599 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:24.905828 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:24.905830 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:24.906010 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:24.906098 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:24.906103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:24.906873 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:24.906875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:24.907508 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:24.907517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:24.908237 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:24.908241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:24.908361 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:24.908876 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:24.908888 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:24.910591 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
W20260619 14:52:24.910867 28771 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:24.905302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:33089 (local address 127.19.34.126:44491)
0619 14:52:24.905472 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:24.905477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:24.905557 (+ 80us) server_negotiation.cc:407] Connection header received
0619 14:52:24.905679 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:24.905681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:24.905725 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:24.905792 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:24.906242 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:24.906781 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:24.907615 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:24.907991 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:24.908106 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:24.910717 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.67:33089: BlockingRecv error: recv got EOF from 127.19.34.67:33089 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 19 14:52:25 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:25 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:25.583067 28774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:25.574294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.65:33071)
0619 14:52:25.574553 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:25.574565 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:25.574647 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:25.574850 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:25.574852 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:25.575007 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:25.575092 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:25.575096 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:25.575716 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:25.575718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:25.576206 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:25.576213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:25.576623 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:25.576626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:25.576688 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:25.581336 (+ 4648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:25.581347 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:25.582923 (+ 1576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":78,"threads_started":1}
W20260619 14:52:25.583191 28775 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:25.574337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:33071 (local address 127.19.34.126:44491)
0619 14:52:25.574550 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:25.574553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:25.574624 (+ 71us) server_negotiation.cc:407] Connection header received
0619 14:52:25.574716 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:25.574718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:25.574756 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:25.574816 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:25.575186 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:25.575638 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:25.576289 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:25.576512 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:25.576587 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:25.583045 (+ 6458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.65:33071: BlockingRecv error: recv got EOF from 127.19.34.65:33071 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
Jun 19 14:52:25 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:25 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:25.737345 28776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:25.732686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.66:50887)
0619 14:52:25.732960 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:25.732972 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:25.733082 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:25.733262 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:25.733264 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:25.733431 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:25.733515 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:25.733520 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:25.734169 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:25.734171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:25.734660 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:25.734666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:25.735011 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:25.735013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:25.735064 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:25.735492 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:25.735502 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:25.737235 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
W20260619 14:52:25.737510 28775 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:25.732710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:50887 (local address 127.19.34.126:44491)
0619 14:52:25.732852 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:25.732855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:25.733052 (+ 197us) server_negotiation.cc:407] Connection header received
0619 14:52:25.733150 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:25.733152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:25.733191 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:25.733252 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:25.733619 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:25.734081 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:25.734742 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:25.734928 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:25.734994 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:25.737358 (+ 2364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.66:50887: BlockingRecv error: recv got EOF from 127.19.34.66:50887 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 14:52:25 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:25 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:25.915916 28777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:25.911369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.67:55135)
0619 14:52:25.911652 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:25.911664 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:25.911750 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:25.911941 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:25.911943 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:25.912114 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:25.912201 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:25.912206 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:25.912857 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:25.912859 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:25.913371 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:25.913378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:25.913730 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:25.913732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:25.913787 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:25.914269 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:25.914279 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:25.915817 (+ 1538us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":79,"threads_started":1}
W20260619 14:52:25.916100 28775 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:25.911437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:55135 (local address 127.19.34.126:44491)
0619 14:52:25.911554 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:25.911557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:25.911708 (+ 151us) server_negotiation.cc:407] Connection header received
0619 14:52:25.911808 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:25.911810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:25.911848 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:25.911906 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:25.912313 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:25.912774 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:25.913459 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:25.913648 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:25.913711 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:25.915929 (+ 2218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.67:55135: BlockingRecv error: recv got EOF from 127.19.34.67:55135 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 14:52:26 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:26 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:26.589119 28778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:26.583779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.65:41323)
0619 14:52:26.584018 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:26.584030 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:26.584108 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:26.584350 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:26.584352 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:26.584513 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:26.584613 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:26.584619 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:26.585324 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:26.585326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:26.586025 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:26.586035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:26.586553 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:26.586557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:26.586664 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:26.587130 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:26.587141 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:26.588961 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
W20260619 14:52:26.589191 28779 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:26.583905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:41323 (local address 127.19.34.126:44491)
0619 14:52:26.584102 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:26.584106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:26.584117 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:52:26.584186 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:26.584188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:26.584224 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:26.584285 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:26.584712 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:26.585242 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:26.586163 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:26.586418 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:26.586558 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:26.589076 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.65:41323: BlockingRecv error: recv got EOF from 127.19.34.65:41323 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":71,"threads_started":1}
Jun 19 14:52:26 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:26 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:26.742977 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:26.738093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.66:53563)
0619 14:52:26.738323 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:26.738335 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:26.738417 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:26.738628 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:26.738630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:26.738785 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:26.738873 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:26.738878 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:26.739664 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:26.739667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:26.740215 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:26.740222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:26.740573 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:26.740575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:26.740626 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:26.741070 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:26.741080 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:26.742857 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
W20260619 14:52:26.743225 28779 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:26.738165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:53563 (local address 127.19.34.126:44491)
0619 14:52:26.738290 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:26.738293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:26.738379 (+ 86us) server_negotiation.cc:407] Connection header received
0619 14:52:26.738494 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:26.738496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:26.738531 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:26.738593 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:26.738963 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:26.739563 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:26.740316 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:26.740494 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:26.740557 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:26.743018 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.66:53563: BlockingRecv error: recv got EOF from 127.19.34.66:53563 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 14:52:26 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:26 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:26.921619 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:26.916709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.67:50215)
0619 14:52:26.916976 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:26.916990 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:26.917083 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:26.917310 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:26.917312 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:26.917484 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:26.917592 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:26.917597 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:26.918290 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:26.918293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:26.918860 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:26.918866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:26.919225 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:26.919227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:26.919281 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:26.919718 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:26.919728 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:26.921472 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":105,"threads_started":1}
W20260619 14:52:26.921908 28779 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:26.916789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:50215 (local address 127.19.34.126:44491)
0619 14:52:26.916923 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:26.916927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:26.917037 (+ 110us) server_negotiation.cc:407] Connection header received
0619 14:52:26.917163 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:26.917165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:26.917206 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:26.917273 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:26.917687 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:26.918195 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:26.918949 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:26.919142 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:26.919208 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:26.921641 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.67:50215: BlockingRecv error: recv got EOF from 127.19.34.67:50215 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 14:52:27 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:27 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:27.594889 28782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:27.589807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.65:33345)
0619 14:52:27.590055 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:27.590072 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:27.590171 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:27.590420 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:27.590423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:27.590629 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:27.590717 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:27.590723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:27.591479 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:27.591484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:27.592035 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:27.592043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:27.592538 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:27.592542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:27.592635 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:27.593079 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:27.593090 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:27.594738 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":87,"threads_started":1}
W20260619 14:52:27.594954 28783 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:27.589912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:33345 (local address 127.19.34.126:44491)
0619 14:52:27.590153 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:27.590157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:27.590169 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:52:27.590262 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:27.590265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:27.590309 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:27.590395 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:27.590837 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:27.591376 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:27.592156 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:27.592431 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:27.592534 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:27.594843 (+ 2309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.65:33345: BlockingRecv error: recv got EOF from 127.19.34.65:33345 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":68,"threads_started":1}
Jun 19 14:52:27 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:27 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:27.748834 28784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:27.743628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.66:38587)
0619 14:52:27.743922 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:27.743935 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:27.744052 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:27.744272 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:27.744275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:27.744473 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:27.744555 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:27.744560 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:27.745241 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:27.745245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:27.745735 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:27.745742 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:27.746149 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:27.746152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:27.746228 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:27.746706 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:27.746720 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:27.748736 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":74,"threads_started":1}
W20260619 14:52:27.748951 28783 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:27.743728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:38587 (local address 127.19.34.126:44491)
0619 14:52:27.743892 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:27.743896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:27.744019 (+ 123us) server_negotiation.cc:407] Connection header received
0619 14:52:27.744126 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:27.744130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:27.744175 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:27.744249 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:27.744670 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:27.745142 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:27.745843 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:27.746043 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:27.746147 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:27.748847 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.66:38587: BlockingRecv error: recv got EOF from 127.19.34.66:38587 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 14:52:27 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:27 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:27.927423 28785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:27.922333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.67:41057)
0619 14:52:27.922582 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:27.922596 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:27.922728 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:27.923154 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:27.923156 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:27.923362 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:27.923447 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:27.923452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:27.924115 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:27.924117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:27.924674 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:27.924682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:27.925042 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:27.925044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:27.925136 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:27.925553 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:27.925563 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:27.927302 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":76,"threads_started":1}
W20260619 14:52:27.927556 28783 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:27.922399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:41057 (local address 127.19.34.126:44491)
0619 14:52:27.922533 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:27.922537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:27.922680 (+ 143us) server_negotiation.cc:407] Connection header received
0619 14:52:27.922839 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:27.922844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:27.922897 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:27.922976 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:27.923541 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:27.924028 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:27.924765 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:27.924957 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:27.925027 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:27.927416 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.67:41057: BlockingRecv error: recv got EOF from 127.19.34.67:41057 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 14:52:28 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:28 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:28.600389 28786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:28.595532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.65:34177)
0619 14:52:28.595774 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:28.595786 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:28.595862 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:28.596070 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:28.596072 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:28.596225 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:28.596459 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:28.596477 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:28.597177 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:28.597179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:28.597739 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:28.597746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:28.598118 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:28.598120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:28.598171 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:28.598656 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:28.598666 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:28.600223 (+ 1557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":89,"threads_started":1}
W20260619 14:52:28.600495 28787 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:28.595589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:34177 (local address 127.19.34.126:44491)
0619 14:52:28.595816 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:28.595819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:28.595830 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:52:28.595938 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:28.595940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:28.595976 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:28.596041 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:28.596559 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:28.597089 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:28.597846 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:28.598039 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:28.598101 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:28.600368 (+ 2267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.65:34177: BlockingRecv error: recv got EOF from 127.19.34.65:34177 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
Jun 19 14:52:28 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:28 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:28.755200 28788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:28.749757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.66:43133)
0619 14:52:28.750044 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:28.750056 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:28.750168 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:28.750413 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:28.750416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:28.750634 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:28.750720 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:28.750725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:28.751642 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:28.751645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:28.752194 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:28.752201 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:28.752658 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:28.752661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:28.752741 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:28.753183 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:28.753193 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:28.755081 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":118,"threads_started":1}
W20260619 14:52:28.755262 28787 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:28.749805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:43133 (local address 127.19.34.126:44491)
0619 14:52:28.749971 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:28.749975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:28.750135 (+ 160us) server_negotiation.cc:407] Connection header received
0619 14:52:28.750266 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:28.750269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:28.750318 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:28.750395 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:28.750831 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:28.751545 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:28.752326 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:28.752540 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:28.752641 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:28.755180 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.66:43133: BlockingRecv error: recv got EOF from 127.19.34.66:43133 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 14:52:28 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:28 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:28.933442 28789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:28.928172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.67:49761)
0619 14:52:28.928456 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:28.928471 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:28.928560 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:28.928854 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:28.928858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:28.929043 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:28.929139 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:28.929146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:28.929881 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:28.929885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:28.930400 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:28.930407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:28.930850 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:28.930854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:28.930942 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:28.931485 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:28.931501 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:28.933342 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":98,"threads_started":1}
W20260619 14:52:28.933561 28787 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:28.928391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:49761 (local address 127.19.34.126:44491)
0619 14:52:28.928506 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:28.928512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:28.928526 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:52:28.928687 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:28.928690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:28.928740 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:28.928825 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:28.929253 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:28.929771 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:28.930507 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:28.930739 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:28.930834 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:28.933456 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.67:49761: BlockingRecv error: recv got EOF from 127.19.34.67:49761 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 19 14:52:29 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:29 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:29.606024 28790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:29.601255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.65:42745)
0619 14:52:29.601488 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:29.601500 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:29.601578 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:29.601794 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:29.601796 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:29.601957 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:29.602041 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:29.602045 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:29.602653 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:29.602655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:29.603138 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:29.603145 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:29.603530 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:29.603532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:29.603585 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:29.604097 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:29.604109 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:29.605927 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":83,"threads_started":1}
W20260619 14:52:29.606182 28791 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:29.601319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:42745 (local address 127.19.34.126:44491)
0619 14:52:29.601524 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:29.601527 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:29.601539 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:52:29.601652 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:29.601654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:29.601692 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:29.601764 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:29.602131 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:29.602576 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:29.603230 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:29.603449 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:29.603513 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:29.606032 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.65:42745: BlockingRecv error: recv got EOF from 127.19.34.65:42745 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":73,"threads_started":1}
Jun 19 14:52:29 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:29 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:29.761431 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:29.755877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.66:55175)
0619 14:52:29.756128 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:29.756141 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:29.756230 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:29.756461 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:29.756463 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:29.756733 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:29.756830 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:29.756837 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:29.757618 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:29.757621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:29.758334 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:29.758343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:29.758790 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:29.758793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:29.758853 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:29.759384 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:29.759395 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:29.761271 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
W20260619 14:52:29.761484 28791 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:29.755952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:55175 (local address 127.19.34.126:44491)
0619 14:52:29.756088 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:29.756091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:29.756204 (+ 113us) server_negotiation.cc:407] Connection header received
0619 14:52:29.756324 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:29.756326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:29.756367 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:29.756426 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:29.756928 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:29.757499 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:29.758435 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:29.758698 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:29.758776 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:29.761380 (+ 2604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.66:55175: BlockingRecv error: recv got EOF from 127.19.34.66:55175 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 14:52:29 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:29 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:29.939358 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:29.934155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.67:44963)
0619 14:52:29.934410 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:29.934423 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:29.934520 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:29.934736 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:29.934738 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:29.934905 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:29.935000 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:29.935005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:29.935708 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:29.935711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:29.936404 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:29.936414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:29.936844 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:29.936847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:29.936911 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:29.937376 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:29.937387 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:29.939206 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":74,"threads_started":1}
W20260619 14:52:29.939401 28791 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:29.934252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:44963 (local address 127.19.34.126:44491)
0619 14:52:29.934399 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:29.934402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:29.934486 (+ 84us) server_negotiation.cc:407] Connection header received
0619 14:52:29.934600 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:29.934602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:29.934642 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:29.934702 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:29.935120 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:29.935594 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:29.936512 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:29.936737 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:29.936817 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:29.939309 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.67:44963: BlockingRecv error: recv got EOF from 127.19.34.67:44963 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 14:52:30 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:30 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:30.619165 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:30.606775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.65:40899)
0619 14:52:30.607053 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:30.607065 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:30.607162 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:30.607386 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:30.607388 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:30.607556 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:30.607651 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:30.607656 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:30.608416 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:30.608418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:30.608947 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:30.608954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:30.609365 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:30.609368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:30.609421 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:30.617249 (+ 7828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:30.617261 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:30.619031 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":106,"threads_started":1}
W20260619 14:52:30.619352 28795 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:30.606828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:40899 (local address 127.19.34.126:44491)
0619 14:52:30.607112 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:30.607116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:30.607130 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:52:30.607232 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:30.607234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:30.607277 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:30.607358 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:30.607748 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:30.608322 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:30.609068 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:30.609274 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:30.609355 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:30.619191 (+ 9836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.65:40899: BlockingRecv error: recv got EOF from 127.19.34.65:40899 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":62,"threads_started":1}
Jun 19 14:52:30 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:30 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:30.766726 28796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:30.762207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.66:44683)
0619 14:52:30.762459 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:30.762470 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:30.762571 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:30.762769 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:30.762771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:30.762923 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:30.763007 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:30.763011 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:30.763627 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:30.763629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:30.764113 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:30.764119 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:30.764470 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:30.764472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:30.764525 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:30.764983 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:30.764994 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:30.766578 (+ 1584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":104,"threads_started":1}
W20260619 14:52:30.766777 28795 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:30.762253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:44683 (local address 127.19.34.126:44491)
0619 14:52:30.762362 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:30.762364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:30.762528 (+ 164us) server_negotiation.cc:407] Connection header received
0619 14:52:30.762645 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:30.762646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:30.762682 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:30.762739 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:30.763109 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:30.763546 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:30.764194 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:30.764387 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:30.764446 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:30.766684 (+ 2238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.66:44683: BlockingRecv error: recv got EOF from 127.19.34.66:44683 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 19 14:52:30 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:30 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:30.945206 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:30.940227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.67:55599)
0619 14:52:30.940555 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:30.940569 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:30.940685 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:30.940928 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:30.940931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:30.941155 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:30.941241 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:30.941246 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:30.941910 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:30.941916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:30.942472 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:30.942479 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:30.942867 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:30.942870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:30.942946 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:30.943411 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:30.943421 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:30.945037 (+ 1616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":119,"threads_started":1}
W20260619 14:52:30.945242 28795 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:30.940329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:55599 (local address 127.19.34.126:44491)
0619 14:52:30.940467 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:30.940472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:30.940638 (+ 166us) server_negotiation.cc:407] Connection header received
0619 14:52:30.940779 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:30.940782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:30.940829 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:30.940908 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:30.941347 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:30.941818 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:30.942576 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:30.942774 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:30.942863 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:30.945137 (+ 2274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.67:55599: BlockingRecv error: recv got EOF from 127.19.34.67:55599 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 19 14:52:31 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:31 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:31.624873 28798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:31.619865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.65:37593)
0619 14:52:31.620117 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:31.620129 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:31.620222 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:31.620490 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:31.620492 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:31.620644 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:31.620726 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:31.620731 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:31.621375 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:31.621377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:31.621930 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:31.621937 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:31.622303 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:31.622305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:31.622358 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:31.622798 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:31.622808 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:31.624727 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":78,"threads_started":1}
W20260619 14:52:31.625010 28799 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:31.619921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:37593 (local address 127.19.34.126:44491)
0619 14:52:31.620133 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:31.620136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:31.620190 (+ 54us) server_negotiation.cc:407] Connection header received
0619 14:52:31.620345 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:31.620351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:31.620397 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:31.620461 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:31.620806 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:31.621289 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:31.622023 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:31.622223 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:31.622289 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:31.624873 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.65:37593: BlockingRecv error: recv got EOF from 127.19.34.65:37593 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":52,"threads_started":1}
Jun 19 14:52:31 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:31 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:31.772486 28800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:31.767463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.66:34931)
0619 14:52:31.767736 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:31.767748 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:31.767847 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:31.768054 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:31.768057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:31.768258 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:31.768381 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:31.768387 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:31.769087 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:31.769089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:31.769774 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:31.769784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:31.770226 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:31.770230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:31.770315 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:31.770733 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:31.770744 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:31.772305 (+ 1561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":113,"threads_started":1}
W20260619 14:52:31.772536 28799 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:31.767566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:34931 (local address 127.19.34.126:44491)
0619 14:52:31.767703 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:31.767706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:31.767820 (+ 114us) server_negotiation.cc:407] Connection header received
0619 14:52:31.767920 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:31.767923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:31.767962 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:31.768030 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:31.768473 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:31.768983 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:31.769875 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:31.770119 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:31.770223 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:31.772438 (+ 2215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.66:34931: BlockingRecv error: recv got EOF from 127.19.34.66:34931 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 14:52:31 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:31 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:31.951603 28801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:31.945896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.67:59563)
0619 14:52:31.946152 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:31.946166 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:31.946266 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:31.946480 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:31.946483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:31.946664 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:31.946761 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:31.946791 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:31.947547 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:31.947550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:31.948259 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:31.948268 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:31.948797 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:31.948801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:31.948879 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:31.949444 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:31.949463 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:31.951427 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":77,"threads_started":1}
W20260619 14:52:31.951642 28799 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:31.946058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:59563 (local address 127.19.34.126:44491)
0619 14:52:31.946181 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:31.946184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:31.946217 (+ 33us) server_negotiation.cc:407] Connection header received
0619 14:52:31.946339 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:31.946341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:31.946381 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:31.946440 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:31.946896 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:31.947437 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:31.948454 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:31.948684 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:31.948796 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:31.951540 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.67:59563: BlockingRecv error: recv got EOF from 127.19.34.67:59563 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 14:52:32 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:32 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:32.630215 28802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:32.625577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.65:44983)
0619 14:52:32.625848 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:32.625860 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:32.625939 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:32.626160 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:32.626162 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:32.626346 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:32.626432 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:32.626436 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:32.627158 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:32.627160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:32.627702 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:32.627708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:32.628095 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:32.628097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:32.628156 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:32.628586 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:32.628596 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:32.630118 (+ 1522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
W20260619 14:52:32.630373 28803 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:32.625635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:44983 (local address 127.19.34.126:44491)
0619 14:52:32.625896 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:32.625899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:32.625910 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:52:32.625997 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:32.625999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:32.626034 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:32.626133 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:32.626518 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:32.627055 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:32.627800 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:32.628009 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:32.628095 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:32.630227 (+ 2132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.65:44983: BlockingRecv error: recv got EOF from 127.19.34.65:44983 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":63,"threads_started":1}
Jun 19 14:52:32 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:32 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:32.778432 28804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:32.773186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.66:38307)
0619 14:52:32.773460 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:32.773471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:32.773583 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:32.773807 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:32.773809 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:32.773984 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:32.774065 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:32.774088 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:32.774742 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:32.774747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:32.775261 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:32.775268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:32.775670 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:32.775673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:32.775750 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:32.776357 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:32.776367 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:32.778293 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":77,"threads_started":1}
W20260619 14:52:32.778468 28803 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:32.773281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:38307 (local address 127.19.34.126:44491)
0619 14:52:32.773422 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:32.773426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:32.773534 (+ 108us) server_negotiation.cc:407] Connection header received
0619 14:52:32.773665 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:32.773669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:32.773716 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:32.773785 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:32.774183 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:32.774642 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:32.775365 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:32.775567 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:32.775668 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:32.778385 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.66:38307: BlockingRecv error: recv got EOF from 127.19.34.66:38307 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 14:52:32 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:32 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.67@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:32.957696 28805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:32.952464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.67:52767)
0619 14:52:32.952747 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:32.952761 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:32.952861 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:32.953088 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:32.953090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:32.953268 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:32.953371 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:32.953377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:32.954081 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:32.954084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:32.954618 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:32.954626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:32.954982 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:32.954985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:32.955050 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:32.955475 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:32.955486 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:32.957513 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":107,"threads_started":1}
W20260619 14:52:32.957791 28803 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:32.952499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:52767 (local address 127.19.34.126:44491)
0619 14:52:32.952624 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:32.952627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:32.952822 (+ 195us) server_negotiation.cc:407] Connection header received
0619 14:52:32.952935 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:32.952937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:32.952978 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:32.953042 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:32.953496 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:32.953984 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:32.954708 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:32.954891 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:32.954955 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:32.957668 (+ 2713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.67:52767: BlockingRecv error: recv got EOF from 127.19.34.67:52767 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 14:52:33 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:33 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.65@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:33.635918 28806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:33.631024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.65:54097)
0619 14:52:33.631254 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:33.631266 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:33.631341 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:33.631535 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:33.631537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:33.631703 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:33.631814 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:33.631820 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:33.632798 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:33.632800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:33.633309 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:33.633316 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:33.633693 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:33.633695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:33.633750 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:33.634244 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:33.634254 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:33.635806 (+ 1552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":74,"threads_started":1}
W20260619 14:52:33.636050 28807 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:33.631100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:54097 (local address 127.19.34.126:44491)
0619 14:52:33.631319 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:33.631323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:33.631334 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:52:33.631402 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:33.631404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:33.631440 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:33.631506 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:33.631927 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:33.632704 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:33.633394 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:33.633590 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:33.633680 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:33.635909 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.65:54097: BlockingRecv error: recv got EOF from 127.19.34.65:54097 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
Jun 19 14:52:33 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:52:33 dist-test-slave-3qdh krb5kdc[27748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.34.66@KRBTEST.COM for oryx/127.19.34.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:52:33.783838 28808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:33.779193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44491 (local address 127.19.34.66:51783)
0619 14:52:33.779418 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:33.779430 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:33.779510 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:33.779711 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:33.779713 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:33.779876 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:33.779961 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:33.779965 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:33.780629 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:33.780631 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:33.781126 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:33.781134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:33.781492 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:33.781494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:33.781547 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:33.782054 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:33.782064 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:33.783719 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.34.126:44491: Server oryx/127.19.34.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":78,"threads_started":1}
W20260619 14:52:33.783987 28807 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:33.779268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:51783 (local address 127.19.34.126:44491)
0619 14:52:33.779376 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:33.779379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:33.779474 (+ 95us) server_negotiation.cc:407] Connection header received
0619 14:52:33.779570 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:33.779571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:33.779618 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:33.779684 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:33.780056 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:33.780541 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:33.781217 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:33.781409 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:33.781479 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:33.783838 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.34.66:51783: BlockingRecv error: recv got EOF from 127.19.34.66:51783 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260619 14:52:33.856592 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 28331
I20260619 14:52:33.861915 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 28483
I20260619 14:52:33.866742 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 28616
I20260619 14:52:33.871493 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 28261
2026-06-19T14:52:33Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17697 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T14:52:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:52:33Z Disabled control of system clock
I20260619 14:52:33.900472 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:38891
--webserver_interface=127.19.34.126
--webserver_port=0
--builtin_ntp_servers=127.19.34.84:44307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:38891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 14:52:34.000566 28812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:34.000792 28812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:34.000841 28812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:34.004807 28812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:34.004856 28812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:34.004873 28812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:34.004889 28812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:34.004904 28812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:52:34.009445 28812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:44307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:38891
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:38891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:34.010281 28812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:34.011457 28812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:34.018973 28817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:34.018992 28820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:34.019058 28818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:34.019225 28812 server_base.cc:1034] running on GCE node
I20260619 14:52:34.019659 28812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:34.020581 28812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:34.021751 28812 hybrid_clock.cc:648] HybridClock initialized: now 1781880754021702 us; error 60 us; skew 500 ppm
I20260619 14:52:34.023438 28812 webserver.cc:460] Webserver started at http://127.19.34.126:41821/ using document root <none> and password file <none>
I20260619 14:52:34.023947 28812 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:34.024024 28812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:34.024219 28812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:34.025964 28812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "85c1bc828ebc41b3a7320a2d01402850"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-3qdh"
server_key: "a90f6e696f8fb084d485b54edc58b2b9"
server_key_iv: "18b1ae994eaa69369d0cd49422542a50"
server_key_version: "encryptionkey@0"
I20260619 14:52:34.026479 28812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "85c1bc828ebc41b3a7320a2d01402850"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-3qdh"
server_key: "a90f6e696f8fb084d485b54edc58b2b9"
server_key_iv: "18b1ae994eaa69369d0cd49422542a50"
server_key_version: "encryptionkey@0"
I20260619 14:52:34.029794 28812 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:52:34.031728 28826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:34.032963 28812 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:34.033078 28812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "85c1bc828ebc41b3a7320a2d01402850"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-3qdh"
server_key: "a90f6e696f8fb084d485b54edc58b2b9"
server_key_iv: "18b1ae994eaa69369d0cd49422542a50"
server_key_version: "encryptionkey@0"
I20260619 14:52:34.033187 28812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:34.041901 28812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:34.042430 28812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:34.042587 28812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:34.048374 28812 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:38891
I20260619 14:52:34.048384 28878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:38891 every 8 connection(s)
I20260619 14:52:34.049157 28812 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:52:34.051805 28879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:34.054965 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 28812
I20260619 14:52:34.055078 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:52:34.055328 19593 external_mini_cluster.cc:1428] Setting key 8325444345a59aaefeaf9f64f6729893
I20260619 14:52:34.056738 28879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850: Bootstrap starting.
I20260619 14:52:34.058646 28879 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:34.059269 28879 log.cc:824] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:34.060833 28879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850: No bootstrap required, opened a new log
I20260619 14:52:34.063176 28882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:34.056524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52512 (local address 127.19.34.126:38891)
0619 14:52:34.057062 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:34.057075 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:34.057106 (+ 31us) server_negotiation.cc:407] Connection header received
0619 14:52:34.057716 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:34.057730 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:34.058060 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:34.058390 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:34.058792 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:34.059668 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:34.060547 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:34.060965 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:34.061098 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:34.061793 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:34.061798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:34.061814 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:34.062123 (+ 309us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:34.062166 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:34.062183 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:34.062273 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:34.062394 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 14:52:34.062618 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":107,"threads_started":1}
I20260619 14:52:34.063902 28879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c1bc828ebc41b3a7320a2d01402850" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38891 } }
I20260619 14:52:34.064073 28879 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:34.064136 28879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85c1bc828ebc41b3a7320a2d01402850, State: Initialized, Role: FOLLOWER
I20260619 14:52:34.064517 28879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c1bc828ebc41b3a7320a2d01402850" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38891 } }
I20260619 14:52:34.064630 28879 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:52:34.064702 28879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:52:34.064805 28879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:34.065604 28879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c1bc828ebc41b3a7320a2d01402850" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38891 } }
I20260619 14:52:34.065891 28879 leader_election.cc:302] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85c1bc828ebc41b3a7320a2d01402850; no voters:
I20260619 14:52:34.066133 28879 leader_election.cc:288] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:52:34.066247 28884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:34.066473 28884 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850 [term 1 LEADER]: Becoming Leader. State: Replica: 85c1bc828ebc41b3a7320a2d01402850, State: Running, Role: LEADER
I20260619 14:52:34.066802 28884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c1bc828ebc41b3a7320a2d01402850" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38891 } }
I20260619 14:52:34.067219 28879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:34.068430 28885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85c1bc828ebc41b3a7320a2d01402850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c1bc828ebc41b3a7320a2d01402850" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38891 } } }
I20260619 14:52:34.068521 28886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85c1bc828ebc41b3a7320a2d01402850. Latest consensus state: current_term: 1 leader_uuid: "85c1bc828ebc41b3a7320a2d01402850" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85c1bc828ebc41b3a7320a2d01402850" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38891 } } }
I20260619 14:52:34.068610 28885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:34.068630 28886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:34.068986 28893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:52:34.071467 28893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:52:34.075627 28893 catalog_manager.cc:1347] Generated new cluster ID: b44a25187c45455a92f71ce4ce01c653
I20260619 14:52:34.075680 28893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:52:34.099273 28893 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:52:34.100353 28893 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:52:34.112998 28893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85c1bc828ebc41b3a7320a2d01402850: Generated new TSK 0
I20260619 14:52:34.113627 28893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:52:34.126569 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:38891
--builtin_ntp_servers=127.19.34.84:44307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 14:52:34.225136 28903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:34.225335 28903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:34.225381 28903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:34.230152 28903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:34.230201 28903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:34.230281 28903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:52:34.234351 28903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:44307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:38891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:34.235208 28903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:34.236404 28903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:34.244624 28908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:34.244592 28911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:34.244657 28909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:34.244993 28903 server_base.cc:1034] running on GCE node
I20260619 14:52:34.245298 28903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:34.245786 28903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:34.246932 28903 hybrid_clock.cc:648] HybridClock initialized: now 1781880754246919 us; error 39 us; skew 500 ppm
I20260619 14:52:34.248522 28903 webserver.cc:460] Webserver started at http://127.19.34.65:39597/ using document root <none> and password file <none>
I20260619 14:52:34.248994 28903 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:34.249050 28903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:34.249202 28903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:34.250823 28903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "da7be457647842fc832ab0306f478926"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-3qdh"
server_key: "fb16c92c781c2255ef225a4ee62c4ae7"
server_key_iv: "0a9b772eeb41eaa01295d4bbe96a4c49"
server_key_version: "encryptionkey@0"
I20260619 14:52:34.251240 28903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "da7be457647842fc832ab0306f478926"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-3qdh"
server_key: "fb16c92c781c2255ef225a4ee62c4ae7"
server_key_iv: "0a9b772eeb41eaa01295d4bbe96a4c49"
server_key_version: "encryptionkey@0"
I20260619 14:52:34.254297 28903 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260619 14:52:34.256372 28917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:34.257472 28903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:52:34.257555 28903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "da7be457647842fc832ab0306f478926"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-3qdh"
server_key: "fb16c92c781c2255ef225a4ee62c4ae7"
server_key_iv: "0a9b772eeb41eaa01295d4bbe96a4c49"
server_key_version: "encryptionkey@0"
I20260619 14:52:34.257674 28903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:34.281457 28903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:34.281980 28903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:34.282146 28903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:34.282618 28903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:34.283349 28903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:34.283387 28903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:34.283440 28903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:34.283466 28903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:34.290845 28903 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:36785
I20260619 14:52:34.290868 29030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:36785 every 8 connection(s)
I20260619 14:52:34.291544 28903 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:52:34.291973 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 28903
I20260619 14:52:34.292074 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:52:34.292387 19593 external_mini_cluster.cc:1428] Setting key d13ce3065236087fc5087064cc0660cd
I20260619 14:52:34.294306 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:38891
--builtin_ntp_servers=127.19.34.84:44307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 14:52:34.301034 28882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:34.293760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:50605 (local address 127.19.34.126:38891)
0619 14:52:34.293856 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:34.293860 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:34.294393 (+ 533us) server_negotiation.cc:407] Connection header received
0619 14:52:34.295732 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:34.295734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:34.295772 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:34.295833 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:34.296836 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:34.297755 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:34.299103 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:34.299504 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:34.299620 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:34.300515 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:34.300521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:34.300527 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:34.300544 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:34.300569 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:34.300582 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:34.300666 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:34.300785 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 14:52:34.300854 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 14:52:34.301693 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:34.293652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:38891 (local address 127.19.34.65:50605)
0619 14:52:34.294275 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:34.294306 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:34.295524 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:34.296017 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:34.296026 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:34.296118 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:34.296711 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:34.296724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:34.297916 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:34.297919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:34.298988 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:34.298997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:34.299635 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:34.299637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:34.299751 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:34.300254 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:52:34.300256 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:34.300287 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:52:34.300382 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:34.300668 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:34.300673 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:34.300676 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:52:34.300757 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:52:34.300932 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":122,"threads_started":1}
I20260619 14:52:34.302767 29031 heartbeater.cc:347] Connected to a master server at 127.19.34.126:38891
I20260619 14:52:34.302955 29031 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:34.303361 29031 heartbeater.cc:511] Master 127.19.34.126:38891 requested a full tablet report, sending...
I20260619 14:52:34.304744 28843 ts_manager.cc:194] Registered new tserver with Master: da7be457647842fc832ab0306f478926 (127.19.34.65:36785)
I20260619 14:52:34.306068 28843 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.34.65:50605
W20260619 14:52:34.393031 29034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:34.393224 29034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:34.393256 29034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:34.396772 29034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:34.396809 29034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:34.396853 29034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:52:34.401222 29034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:44307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:38891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:34.401986 29034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:34.402993 29034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:34.411195 29040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:34.411249 29034 server_base.cc:1034] running on GCE node
W20260619 14:52:34.411206 29039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:34.411206 29042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:34.411779 29034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:34.412281 29034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:34.413476 29034 hybrid_clock.cc:648] HybridClock initialized: now 1781880754413454 us; error 34 us; skew 500 ppm
I20260619 14:52:34.415004 29034 webserver.cc:460] Webserver started at http://127.19.34.66:41621/ using document root <none> and password file <none>
I20260619 14:52:34.415493 29034 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:34.415570 29034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:34.415756 29034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:34.417635 29034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "311fba5214114fb9a6fcc4505d3adb61"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-3qdh"
server_key: "23e2c95ad6e445b3a73352a7c654c9d8"
server_key_iv: "22991f9206212fd189e5c8ce43ed1962"
server_key_version: "encryptionkey@0"
I20260619 14:52:34.418143 29034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "311fba5214114fb9a6fcc4505d3adb61"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-3qdh"
server_key: "23e2c95ad6e445b3a73352a7c654c9d8"
server_key_iv: "22991f9206212fd189e5c8ce43ed1962"
server_key_version: "encryptionkey@0"
I20260619 14:52:34.421342 29034 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 14:52:34.423672 29048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:34.424882 29034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:52:34.425001 29034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "311fba5214114fb9a6fcc4505d3adb61"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-3qdh"
server_key: "23e2c95ad6e445b3a73352a7c654c9d8"
server_key_iv: "22991f9206212fd189e5c8ce43ed1962"
server_key_version: "encryptionkey@0"
I20260619 14:52:34.425115 29034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:34.450824 29034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:34.451345 29034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:34.451499 29034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:34.451946 29034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:34.452796 29034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:34.452860 29034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:34.452921 29034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:34.452963 29034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:34.460799 29034 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:46591
I20260619 14:52:34.460839 29161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:46591 every 8 connection(s)
I20260619 14:52:34.461565 29034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:52:34.469575 28882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:34.463163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:32787 (local address 127.19.34.126:38891)
0619 14:52:34.463297 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:34.463301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:34.464247 (+ 946us) server_negotiation.cc:407] Connection header received
0619 14:52:34.465378 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:34.465381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:34.465429 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:34.465507 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:34.466245 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:34.466728 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:34.467784 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:34.468115 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:34.468208 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:34.469182 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:34.469184 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:34.469186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:34.469197 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:34.469215 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:34.469223 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:34.469278 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:34.469391 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 14:52:34.469457 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:52:34.469982 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 29034
I20260619 14:52:34.470073 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:52:34.470263 29164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:34.463511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:38891 (local address 127.19.34.66:32787)
0619 14:52:34.464098 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:34.464139 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:34.465211 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:34.465630 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:34.465635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:34.465704 (+ 69us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:34.466093 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:34.466103 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:34.466817 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:34.466820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:34.467658 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:34.467666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:34.468243 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:34.468246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:34.468457 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:34.468955 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:52:34.468957 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:34.469000 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:52:34.469049 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:34.469285 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:34.469291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:34.469294 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:52:34.469369 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 14:52:34.469566 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":116,"threads_started":1}
I20260619 14:52:34.470322 19593 external_mini_cluster.cc:1428] Setting key 09c8e370fcce6f998d19788dec7ee3f2
I20260619 14:52:34.471315 29162 heartbeater.cc:347] Connected to a master server at 127.19.34.126:38891
I20260619 14:52:34.471542 29162 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:34.471999 29162 heartbeater.cc:511] Master 127.19.34.126:38891 requested a full tablet report, sending...
I20260619 14:52:34.472167 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:38891
--builtin_ntp_servers=127.19.34.84:44307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 14:52:34.473054 28843 ts_manager.cc:194] Registered new tserver with Master: 311fba5214114fb9a6fcc4505d3adb61 (127.19.34.66:46591)
I20260619 14:52:34.473845 28843 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.34.66:32787
W20260619 14:52:34.570628 29165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:34.570823 29165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:34.570853 29165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:34.574234 29165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:34.574272 29165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:34.574313 29165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:52:34.578390 29165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:44307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:38891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:34.579449 29165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:34.580829 29165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:34.588580 29170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:34.588585 29171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:34.588755 29165 server_base.cc:1034] running on GCE node
W20260619 14:52:34.588620 29173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:34.589175 29165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:34.589644 29165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:34.590804 29165 hybrid_clock.cc:648] HybridClock initialized: now 1781880754590781 us; error 32 us; skew 500 ppm
I20260619 14:52:34.592273 29165 webserver.cc:460] Webserver started at http://127.19.34.67:41343/ using document root <none> and password file <none>
I20260619 14:52:34.592777 29165 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:34.592829 29165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:34.593029 29165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:34.594622 29165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "93347a0ccc5b4664a8bfda513ac4a540"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-3qdh"
server_key: "1a6c0625f9c0217e191a075f4c129893"
server_key_iv: "c10afb63e07eeb808ccb83fef0d034bb"
server_key_version: "encryptionkey@0"
I20260619 14:52:34.595075 29165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "93347a0ccc5b4664a8bfda513ac4a540"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-3qdh"
server_key: "1a6c0625f9c0217e191a075f4c129893"
server_key_iv: "c10afb63e07eeb808ccb83fef0d034bb"
server_key_version: "encryptionkey@0"
I20260619 14:52:34.598031 29165 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:52:34.600061 29179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:34.601066 29165 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:34.601162 29165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "93347a0ccc5b4664a8bfda513ac4a540"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-3qdh"
server_key: "1a6c0625f9c0217e191a075f4c129893"
server_key_iv: "c10afb63e07eeb808ccb83fef0d034bb"
server_key_version: "encryptionkey@0"
I20260619 14:52:34.601243 29165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:34.611186 29165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:34.611657 29165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:34.611807 29165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:34.612251 29165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:34.613104 29165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:34.613159 29165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:34.613209 29165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:34.613242 29165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:34.620903 29165 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:46367
I20260619 14:52:34.620929 29292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:46367 every 8 connection(s)
I20260619 14:52:34.621632 29165 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:52:34.628716 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 29165
I20260619 14:52:34.628813 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:52:34.629046 19593 external_mini_cluster.cc:1428] Setting key 30462c0fd3ea0b5433302d756638b2b9
I20260619 14:52:34.629624 28882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:34.623087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:41415 (local address 127.19.34.126:38891)
0619 14:52:34.623211 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:34.623217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:34.624163 (+ 946us) server_negotiation.cc:407] Connection header received
0619 14:52:34.625506 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:34.625511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:34.625567 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:34.625633 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:34.626341 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:34.626798 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:34.627851 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:34.628061 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:34.628138 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:34.629204 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:34.629212 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:34.629213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:34.629223 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:34.629239 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:34.629246 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:34.629309 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:34.629447 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 14:52:34.629498 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 14:52:34.630440 29295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:34.623406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:38891 (local address 127.19.34.67:41415)
0619 14:52:34.624000 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:34.624033 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:34.625323 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:34.625749 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:34.625754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:34.625822 (+ 68us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:34.626242 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:34.626252 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:34.626898 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:34.626900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:34.627734 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:34.627742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:34.628172 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:34.628181 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:34.628381 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:34.628967 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:52:34.628968 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:34.628999 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:52:34.629049 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:34.629322 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:34.629330 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:34.629334 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:52:34.629446 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 14:52:34.629672 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":89,"threads_started":1}
I20260619 14:52:34.631408 29293 heartbeater.cc:347] Connected to a master server at 127.19.34.126:38891
I20260619 14:52:34.631597 29293 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:34.631996 29293 heartbeater.cc:511] Master 127.19.34.126:38891 requested a full tablet report, sending...
I20260619 14:52:34.632882 28843 ts_manager.cc:194] Registered new tserver with Master: 93347a0ccc5b4664a8bfda513ac4a540 (127.19.34.67:46367)
I20260619 14:52:34.633438 28843 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.34.67:41415
I20260619 14:52:34.642343 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 14:52:34.644591 28882 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:34.643811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52514 (local address 127.19.34.126:38891)
0619 14:52:34.643904 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:34.643909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:34.644077 (+ 168us) server_negotiation.cc:407] Connection header received
0619 14:52:34.644176 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:34.644178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:34.644214 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:34.644278 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:34.644499 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52514: BlockingRecv error: recv got EOF from 127.0.0.1:52514 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 14:52:34.644726 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 28903
I20260619 14:52:34.649541 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 29034
I20260619 14:52:34.654255 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 29165
I20260619 14:52:34.658720 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 28812
2026-06-19T14:52:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (784 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T14:52:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:52:34Z Disabled control of system clock
I20260619 14:52:34.684115 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:43337
--webserver_interface=127.19.34.126
--webserver_port=0
--builtin_ntp_servers=127.19.34.84:33057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:43337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 14:52:34.779282 29304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:34.779476 29304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:34.779507 29304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:34.783120 29304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:34.783164 29304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:34.783178 29304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:34.783226 29304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:34.783258 29304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:52:34.787402 29304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:33057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:43337
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:43337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:34.788264 29304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:34.789438 29304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:34.796366 29310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:34.796397 29312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:34.796480 29304 server_base.cc:1034] running on GCE node
W20260619 14:52:34.796429 29309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:34.796936 29304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:34.797901 29304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:34.799093 29304 hybrid_clock.cc:648] HybridClock initialized: now 1781880754799067 us; error 25 us; skew 500 ppm
I20260619 14:52:34.800679 29304 webserver.cc:460] Webserver started at http://127.19.34.126:42429/ using document root <none> and password file <none>
I20260619 14:52:34.801182 29304 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:34.801337 29304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:34.801543 29304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:34.803179 29304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "e8f556314f8f403180345e107ce36854"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-3qdh"
server_key: "4b954aeb0001ea2c11482f2bb9dd004d"
server_key_iv: "9690b6538f41e77e55818fdfcecdfc9d"
server_key_version: "encryptionkey@0"
I20260619 14:52:34.803639 29304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "e8f556314f8f403180345e107ce36854"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-3qdh"
server_key: "4b954aeb0001ea2c11482f2bb9dd004d"
server_key_iv: "9690b6538f41e77e55818fdfcecdfc9d"
server_key_version: "encryptionkey@0"
I20260619 14:52:34.806595 29304 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:52:34.808667 29318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:34.809618 29304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:34.809732 29304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "e8f556314f8f403180345e107ce36854"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-3qdh"
server_key: "4b954aeb0001ea2c11482f2bb9dd004d"
server_key_iv: "9690b6538f41e77e55818fdfcecdfc9d"
server_key_version: "encryptionkey@0"
I20260619 14:52:34.809820 29304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:34.813635 29304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:34.814129 29304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:34.814280 29304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:34.821501 29304 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:43337
I20260619 14:52:34.821504 29370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:43337 every 8 connection(s)
I20260619 14:52:34.822247 29304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:52:34.824926 29371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:34.828267 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 29304
I20260619 14:52:34.828451 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:52:34.828689 19593 external_mini_cluster.cc:1428] Setting key 61bf60c12a2bc0063b62050193f72a67
I20260619 14:52:34.829644 29371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854: Bootstrap starting.
I20260619 14:52:34.831475 29371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:34.832288 29371 log.cc:824] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:34.832995 29374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:34.829776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55686 (local address 127.19.34.126:43337)
0619 14:52:34.830309 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:34.830323 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:34.830352 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:52:34.830857 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:34.830869 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:34.831175 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:34.831452 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:34.832003 (+ 551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:34.832006 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:34.832015 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:34.832022 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:34.832061 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:34.832089 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:34.832188 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:34.832273 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 14:52:34.832495 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":108,"threads_started":1}
I20260619 14:52:34.833827 29371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854: No bootstrap required, opened a new log
I20260619 14:52:34.836740 29371 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f556314f8f403180345e107ce36854" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43337 } }
I20260619 14:52:34.836903 29371 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:34.836936 29371 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f556314f8f403180345e107ce36854, State: Initialized, Role: FOLLOWER
I20260619 14:52:34.837289 29371 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f556314f8f403180345e107ce36854" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43337 } }
I20260619 14:52:34.837399 29371 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:52:34.837441 29371 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:52:34.837505 29371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:34.838245 29371 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f556314f8f403180345e107ce36854" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43337 } }
I20260619 14:52:34.838536 29371 leader_election.cc:302] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8f556314f8f403180345e107ce36854; no voters:
I20260619 14:52:34.838723 29371 leader_election.cc:288] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:52:34.838819 29376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:34.838997 29376 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854 [term 1 LEADER]: Becoming Leader. State: Replica: e8f556314f8f403180345e107ce36854, State: Running, Role: LEADER
I20260619 14:52:34.839339 29376 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f556314f8f403180345e107ce36854" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43337 } }
I20260619 14:52:34.839772 29371 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:34.840652 29377 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8f556314f8f403180345e107ce36854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f556314f8f403180345e107ce36854" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43337 } } }
I20260619 14:52:34.840763 29378 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8f556314f8f403180345e107ce36854. Latest consensus state: current_term: 1 leader_uuid: "e8f556314f8f403180345e107ce36854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f556314f8f403180345e107ce36854" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43337 } } }
I20260619 14:52:34.840768 29377 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:34.840910 29378 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:34.841238 29385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:52:34.843463 29385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:52:34.848080 29385 catalog_manager.cc:1347] Generated new cluster ID: 6cc6d565d9d647aba05a47bbada96868
I20260619 14:52:34.848135 29385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:52:34.854598 29385 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:52:34.854840 29385 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:52:34.861562 29385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8f556314f8f403180345e107ce36854: Generated new TSK 0
I20260619 14:52:34.862113 29385 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:52:34.868148 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:43337
--builtin_ntp_servers=127.19.34.84:33057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 14:52:34.973779 29395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:34.974025 29395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:34.974073 29395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:34.978231 29395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:34.978291 29395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:34.978348 29395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:52:34.982961 29395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:33057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:43337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:34.983929 29395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:34.985251 29395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:34.994594 29403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:34.994602 29401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:34.994570 29400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:34.994750 29395 server_base.cc:1034] running on GCE node
I20260619 14:52:34.995191 29395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:34.995743 29395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:34.996927 29395 hybrid_clock.cc:648] HybridClock initialized: now 1781880754996913 us; error 24 us; skew 500 ppm
I20260619 14:52:34.998634 29395 webserver.cc:460] Webserver started at http://127.19.34.65:43945/ using document root <none> and password file <none>
I20260619 14:52:34.999163 29395 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:34.999248 29395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:34.999457 29395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:35.001436 29395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "46a9073163504c23a15502ec73f74164"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-3qdh"
server_key: "f06d4058e21bd90f01d939118402b321"
server_key_iv: "c3eba22f98e8977c9fb64128a9e97379"
server_key_version: "encryptionkey@0"
I20260619 14:52:35.001959 29395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "46a9073163504c23a15502ec73f74164"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-3qdh"
server_key: "f06d4058e21bd90f01d939118402b321"
server_key_iv: "c3eba22f98e8977c9fb64128a9e97379"
server_key_version: "encryptionkey@0"
I20260619 14:52:35.005318 29395 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:52:35.008025 29409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:35.009150 29395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:52:35.009268 29395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "46a9073163504c23a15502ec73f74164"
format_stamp: "Formatted at 2026-06-19 14:52:34 on dist-test-slave-3qdh"
server_key: "f06d4058e21bd90f01d939118402b321"
server_key_iv: "c3eba22f98e8977c9fb64128a9e97379"
server_key_version: "encryptionkey@0"
I20260619 14:52:35.009370 29395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:35.013970 29395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:35.014545 29395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:35.014707 29395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:35.015192 29395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:35.016039 29395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:35.016111 29395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:35.016191 29395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:35.016229 29395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:35.025476 29395 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:34517
I20260619 14:52:35.025496 29522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:34517 every 8 connection(s)
I20260619 14:52:35.026283 29395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:52:35.032001 29374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:35.027938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:36473 (local address 127.19.34.126:43337)
0619 14:52:35.028096 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:35.028100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:35.029115 (+ 1015us) server_negotiation.cc:407] Connection header received
0619 14:52:35.030364 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:35.030367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:35.030423 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:35.030519 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:35.031545 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:35.031549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:35.031551 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:35.031566 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:35.031588 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:35.031601 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:35.031689 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:35.031829 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 14:52:35.031872 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:52:35.032843 29525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:35.028291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43337 (local address 127.19.34.65:36473)
0619 14:52:35.028961 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:35.028996 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:35.030163 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:35.030672 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:35.030678 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:35.030757 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:35.031261 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:52:35.031262 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:35.031297 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:52:35.031356 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:35.031684 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:35.031693 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:35.031701 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:52:35.031871 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 14:52:35.032086 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":119,"threads_started":1}
I20260619 14:52:35.033638 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 29395
I20260619 14:52:35.033759 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:52:35.034024 29523 heartbeater.cc:347] Connected to a master server at 127.19.34.126:43337
I20260619 14:52:35.034068 19593 external_mini_cluster.cc:1428] Setting key da476a72c831f3252bf3133bae28990b
I20260619 14:52:35.034228 29523 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:35.034597 29523 heartbeater.cc:511] Master 127.19.34.126:43337 requested a full tablet report, sending...
I20260619 14:52:35.035749 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:43337
--builtin_ntp_servers=127.19.34.84:33057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 14:52:35.035920 29327 ts_manager.cc:194] Registered new tserver with Master: 46a9073163504c23a15502ec73f74164 (127.19.34.65:34517)
W20260619 14:52:35.140995 29526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:35.141207 29526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:35.141243 29526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:35.145220 29526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:35.145263 29526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:35.145337 29526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:52:35.149811 29526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:33057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:43337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:35.150688 29526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:35.151892 29526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:35.160652 29534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:35.160748 29531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:35.160759 29526 server_base.cc:1034] running on GCE node
W20260619 14:52:35.160807 29532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:35.161338 29526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:35.161870 29526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:35.163019 29526 hybrid_clock.cc:648] HybridClock initialized: now 1781880755162998 us; error 33 us; skew 500 ppm
I20260619 14:52:35.164708 29526 webserver.cc:460] Webserver started at http://127.19.34.66:44119/ using document root <none> and password file <none>
I20260619 14:52:35.165215 29526 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:35.165272 29526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:35.165483 29526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:35.167352 29526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "e5c269bcf8da445e9596cb5e777b8837"
format_stamp: "Formatted at 2026-06-19 14:52:35 on dist-test-slave-3qdh"
server_key: "b873bd59bc13948fa3004e9eeecec47c"
server_key_iv: "ca2b54a0a05dccda96924c9327019b73"
server_key_version: "encryptionkey@0"
I20260619 14:52:35.167877 29526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "e5c269bcf8da445e9596cb5e777b8837"
format_stamp: "Formatted at 2026-06-19 14:52:35 on dist-test-slave-3qdh"
server_key: "b873bd59bc13948fa3004e9eeecec47c"
server_key_iv: "ca2b54a0a05dccda96924c9327019b73"
server_key_version: "encryptionkey@0"
I20260619 14:52:35.171319 29526 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:52:35.173769 29540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:35.174868 29526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:52:35.175004 29526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "e5c269bcf8da445e9596cb5e777b8837"
format_stamp: "Formatted at 2026-06-19 14:52:35 on dist-test-slave-3qdh"
server_key: "b873bd59bc13948fa3004e9eeecec47c"
server_key_iv: "ca2b54a0a05dccda96924c9327019b73"
server_key_version: "encryptionkey@0"
I20260619 14:52:35.175118 29526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:35.179498 29526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:35.180059 29526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:35.180236 29526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:35.180751 29526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:35.181597 29526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:35.181663 29526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:35.181725 29526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:35.181770 29526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:35.190476 29526 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:40347
I20260619 14:52:35.190495 29653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:40347 every 8 connection(s)
I20260619 14:52:35.191687 29526 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:52:35.197196 29374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:35.193445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:34739 (local address 127.19.34.126:43337)
0619 14:52:35.193590 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:35.193595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:35.194539 (+ 944us) server_negotiation.cc:407] Connection header received
0619 14:52:35.195744 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:35.195746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:35.195781 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:35.195852 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:35.196840 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:35.196843 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:35.196845 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:35.196855 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:35.196874 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:35.196881 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:35.196950 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:35.197055 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 14:52:35.197088 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:52:35.197942 29656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:35.193796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43337 (local address 127.19.34.66:34739)
0619 14:52:35.194391 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:35.194425 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:35.195574 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:35.196006 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:35.196012 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:35.196109 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:35.196620 (+ 511us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:52:35.196622 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:35.196656 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:52:35.196716 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:35.196947 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:35.196953 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:35.196956 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:52:35.197088 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 14:52:35.197272 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":117,"threads_started":1}
I20260619 14:52:35.199096 29654 heartbeater.cc:347] Connected to a master server at 127.19.34.126:43337
I20260619 14:52:35.199295 29654 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:35.199641 29654 heartbeater.cc:511] Master 127.19.34.126:43337 requested a full tablet report, sending...
I20260619 14:52:35.200601 29327 ts_manager.cc:194] Registered new tserver with Master: e5c269bcf8da445e9596cb5e777b8837 (127.19.34.66:40347)
I20260619 14:52:35.201344 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 29526
I20260619 14:52:35.201460 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:52:35.201713 19593 external_mini_cluster.cc:1428] Setting key 925997739639bea5892a64b4c4e4ee56
I20260619 14:52:35.203366 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:43337
--builtin_ntp_servers=127.19.34.84:33057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 14:52:35.307900 29657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:35.308116 29657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:35.308153 29657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:35.312237 29657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:35.312281 29657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:35.312412 29657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:52:35.317270 29657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:33057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:43337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:35.318130 29657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:35.319366 29657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:35.328188 29663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:35.328154 29662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:35.328230 29665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:35.328347 29657 server_base.cc:1034] running on GCE node
I20260619 14:52:35.328729 29657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:35.329277 29657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:35.330448 29657 hybrid_clock.cc:648] HybridClock initialized: now 1781880755330422 us; error 45 us; skew 500 ppm
I20260619 14:52:35.332023 29657 webserver.cc:460] Webserver started at http://127.19.34.67:38973/ using document root <none> and password file <none>
I20260619 14:52:35.332707 29657 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:35.332813 29657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:35.332998 29657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:35.334756 29657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "01e8cb9e02554ab6af7eebbb088f8ded"
format_stamp: "Formatted at 2026-06-19 14:52:35 on dist-test-slave-3qdh"
server_key: "49c2761294927d5b689f3e9ea6d33d2d"
server_key_iv: "486cf11d793cd52628b3102cdcab7f42"
server_key_version: "encryptionkey@0"
I20260619 14:52:35.335255 29657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "01e8cb9e02554ab6af7eebbb088f8ded"
format_stamp: "Formatted at 2026-06-19 14:52:35 on dist-test-slave-3qdh"
server_key: "49c2761294927d5b689f3e9ea6d33d2d"
server_key_iv: "486cf11d793cd52628b3102cdcab7f42"
server_key_version: "encryptionkey@0"
I20260619 14:52:35.338580 29657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 14:52:35.340881 29671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:35.342020 29657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:52:35.342103 29657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "01e8cb9e02554ab6af7eebbb088f8ded"
format_stamp: "Formatted at 2026-06-19 14:52:35 on dist-test-slave-3qdh"
server_key: "49c2761294927d5b689f3e9ea6d33d2d"
server_key_iv: "486cf11d793cd52628b3102cdcab7f42"
server_key_version: "encryptionkey@0"
I20260619 14:52:35.342182 29657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:35.346500 29657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:35.346990 29657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:35.347127 29657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:35.347568 29657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:35.348433 29657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:35.348477 29657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:35.348513 29657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:35.348534 29657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:35.356117 29657 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:37935
I20260619 14:52:35.356271 29784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:37935 every 8 connection(s)
I20260619 14:52:35.356976 29657 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:52:35.358429 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 29657
I20260619 14:52:35.358532 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:52:35.358829 19593 external_mini_cluster.cc:1428] Setting key 63e85c38beb8577142b514b48cf91707
I20260619 14:52:35.362391 29374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:35.358531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:43759 (local address 127.19.34.126:43337)
0619 14:52:35.358660 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:35.358665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:35.359684 (+ 1019us) server_negotiation.cc:407] Connection header received
0619 14:52:35.360966 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:35.360969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:35.361007 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:35.361092 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:35.362087 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:35.362089 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:35.362091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:52:35.362102 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:35.362120 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:52:35.362128 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:35.362193 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:35.362262 (+ 69us) server_negotiation.cc:299] Negotiation successful
0619 14:52:35.362292 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 14:52:35.363140 29787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:35.358903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43337 (local address 127.19.34.67:43759)
0619 14:52:35.359550 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:35.359595 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:35.360788 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:35.361257 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:35.361263 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:35.361342 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:35.361853 (+ 511us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:52:35.361855 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:35.361888 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:52:35.361946 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:35.362183 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:35.362186 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:35.362189 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:52:35.362293 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 14:52:35.362448 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":119,"threads_started":1}
I20260619 14:52:35.364101 29785 heartbeater.cc:347] Connected to a master server at 127.19.34.126:43337
I20260619 14:52:35.364346 29785 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:35.364682 29785 heartbeater.cc:511] Master 127.19.34.126:43337 requested a full tablet report, sending...
I20260619 14:52:35.365547 29327 ts_manager.cc:194] Registered new tserver with Master: 01e8cb9e02554ab6af7eebbb088f8ded (127.19.34.67:37935)
I20260619 14:52:35.372102 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 14:52:35.374629 29374 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:52:35.373886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55688 (local address 127.19.34.126:43337)
0619 14:52:35.373983 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:35.373986 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:35.374185 (+ 199us) server_negotiation.cc:407] Connection header received
0619 14:52:35.374262 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:35.374264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:35.374289 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:35.374349 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:35.374505 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55688: BlockingRecv error: recv got EOF from 127.0.0.1:55688 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:52:35.374890 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 29395
I20260619 14:52:35.379703 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 29526
I20260619 14:52:35.384728 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 29657
I20260619 14:52:35.389511 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 29304
2026-06-19T14:52:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (731 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:52:35 dist-test-slave-3qdh krb5kdc[29795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:52:35 dist-test-slave-3qdh krb5kdc[29795](info): set up 2 sockets
Jun 19 14:52:35 dist-test-slave-3qdh krb5kdc[29795](info): commencing operation
krb5kdc: starting...
W20260619 14:52:36.376821 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:52:36 dist-test-slave-3qdh krb5kdc[29795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:52:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:52:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:52:36.483048 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:36295
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:36295
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:36.580287 29808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:36.580489 29808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:36.580523 29808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:36.584012 29808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:52:36.584050 29808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:36.584066 29808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:36.584080 29808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:52:36.584092 29808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:52:36.588591 29808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:36295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:36295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:36.589345 29808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:36.590085 29808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:36.597064 29813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:36.597103 29816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:36.597055 29814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:36.597110 29808 server_base.cc:1034] running on GCE node
I20260619 14:52:36.597797 29808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:36.598757 29808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:36.599943 29808 hybrid_clock.cc:648] HybridClock initialized: now 1781880756599927 us; error 30 us; skew 500 ppm
Jun 19 14:52:36 dist-test-slave-3qdh krb5kdc[29795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880756, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:36.608189 29808 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:52:36.608989 29808 webserver.cc:460] Webserver started at http://127.19.34.126:46539/ using document root <none> and password file <none>
I20260619 14:52:36.609403 29808 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:36.609493 29808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:36.609642 29808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:36.611246 29808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "e15b567e968c48cb9e17accc7da25724"
format_stamp: "Formatted at 2026-06-19 14:52:36 on dist-test-slave-3qdh"
server_key: "dd603cebfa9b67906d1a2583bc1e8f22"
server_key_iv: "1b8b97c08a31f50b80127cfa924ffdb8"
server_key_version: "encryptionkey@0"
I20260619 14:52:36.611681 29808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "e15b567e968c48cb9e17accc7da25724"
format_stamp: "Formatted at 2026-06-19 14:52:36 on dist-test-slave-3qdh"
server_key: "dd603cebfa9b67906d1a2583bc1e8f22"
server_key_iv: "1b8b97c08a31f50b80127cfa924ffdb8"
server_key_version: "encryptionkey@0"
I20260619 14:52:36.614817 29808 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 14:52:36.616780 29823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:36.617842 29808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:36.617956 29808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "e15b567e968c48cb9e17accc7da25724"
format_stamp: "Formatted at 2026-06-19 14:52:36 on dist-test-slave-3qdh"
server_key: "dd603cebfa9b67906d1a2583bc1e8f22"
server_key_iv: "1b8b97c08a31f50b80127cfa924ffdb8"
server_key_version: "encryptionkey@0"
I20260619 14:52:36.618045 29808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:36.641002 29808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:36.643282 29808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:36.643458 29808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:36.649480 29808 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:36295
I20260619 14:52:36.649727 29875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:36295 every 8 connection(s)
I20260619 14:52:36.650251 29808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:52:36.652603 29876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:36.657217 29876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724: Bootstrap starting.
I20260619 14:52:36.657820 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 29808
I20260619 14:52:36.657945 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:52:36.658201 19593 external_mini_cluster.cc:1428] Setting key f74a16c1d0b14dba47300fa99634a508
I20260619 14:52:36.659616 29876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:36.660192 29876 log.cc:824] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:36.661635 29876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724: No bootstrap required, opened a new log
I20260619 14:52:36.664417 29876 raft_consensus.cc:348] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15b567e968c48cb9e17accc7da25724" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 36295 } }
I20260619 14:52:36.664633 29876 raft_consensus.cc:374] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:36.664696 29876 raft_consensus.cc:729] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e15b567e968c48cb9e17accc7da25724, State: Initialized, Role: FOLLOWER
I20260619 14:52:36.665125 29876 consensus_queue.cc:260] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15b567e968c48cb9e17accc7da25724" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 36295 } }
I20260619 14:52:36.665272 29876 raft_consensus.cc:388] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:52:36.665331 29876 raft_consensus.cc:482] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Jun 19 14:52:36 dist-test-slave-3qdh krb5kdc[29795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:52:36.665417 29876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:36.666133 29876 raft_consensus.cc:504] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15b567e968c48cb9e17accc7da25724" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 36295 } }
I20260619 14:52:36.666414 29876 leader_election.cc:302] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e15b567e968c48cb9e17accc7da25724; no voters:
I20260619 14:52:36.666671 29876 leader_election.cc:288] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:52:36.666978 29881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:36.667213 29881 raft_consensus.cc:686] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724 [term 1 LEADER]: Becoming Leader. State: Replica: e15b567e968c48cb9e17accc7da25724, State: Running, Role: LEADER
I20260619 14:52:36.667524 29881 consensus_queue.cc:237] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15b567e968c48cb9e17accc7da25724" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 36295 } }
I20260619 14:52:36.667829 29876 sys_catalog.cc:564] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:52:36.669047 29882 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e15b567e968c48cb9e17accc7da25724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15b567e968c48cb9e17accc7da25724" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 36295 } } }
I20260619 14:52:36.669162 29882 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:36.669503 29883 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e15b567e968c48cb9e17accc7da25724. Latest consensus state: current_term: 1 leader_uuid: "e15b567e968c48cb9e17accc7da25724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e15b567e968c48cb9e17accc7da25724" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 36295 } } }
I20260619 14:52:36.669642 29883 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724 [sys.catalog]: This master's current role is: LEADER
I20260619 14:52:36.670197 29891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:52:36.670933 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:36.659195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52152 (local address 127.19.34.126:36295)
0619 14:52:36.659516 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:36.659527 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:36.659556 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:52:36.660103 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:36.660116 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:36.660427 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:36.660697 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:36.661370 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:36.662217 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:36.662972 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:36.663389 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:36.663540 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:36.665887 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:36.665892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:36.665908 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:36.665929 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:36.667764 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:36.668248 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:36.668254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:36.668259 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:36.668415 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:36.668763 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:36.668768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:36.668770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:36.669235 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:36.669421 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:36.670122 (+ 701us) server_negotiation.cc:299] Negotiation successful
0619 14:52:36.670359 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":100,"threads_started":1}
I20260619 14:52:36.673045 29891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:52:36.677454 29891 catalog_manager.cc:1347] Generated new cluster ID: f988883a59a849d7a9e5e8fe8d534fc6
I20260619 14:52:36.677513 29891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:52:36.684391 29891 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:52:36.685371 29891 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:52:36.692008 29891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724: Generated new TSK 0
I20260619 14:52:36.692584 29891 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.34.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
I20260619 14:52:36.725292 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:36295
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:36.848047 29902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:36.848297 29902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:36.848415 29902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:36.851976 29902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:36.852051 29902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:36.852134 29902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:52:36.856535 29902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:36295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:36.857429 29902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:36.858305 29902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:36.866544 29908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:36.866559 29907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:36.866688 29902 server_base.cc:1034] running on GCE node
W20260619 14:52:36.866744 29910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:36.867093 29902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:36.867676 29902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:36.868855 29902 hybrid_clock.cc:648] HybridClock initialized: now 1781880756868831 us; error 29 us; skew 500 ppm
Jun 19 14:52:36 dist-test-slave-3qdh krb5kdc[29795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880756, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:36.871421 29902 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:52:36.872148 29902 webserver.cc:460] Webserver started at http://127.19.34.65:38497/ using document root <none> and password file <none>
I20260619 14:52:36.872650 29902 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:36.872722 29902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:36.872886 29902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:36.874418 29902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "a58b25f56b454a3cab58087f64d48206"
format_stamp: "Formatted at 2026-06-19 14:52:36 on dist-test-slave-3qdh"
server_key: "1af8cf18e1613538e8adc15a6e39d4c3"
server_key_iv: "a3085b0d7a75384c4b6aead68e2bf6c6"
server_key_version: "encryptionkey@0"
I20260619 14:52:36.874888 29902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "a58b25f56b454a3cab58087f64d48206"
format_stamp: "Formatted at 2026-06-19 14:52:36 on dist-test-slave-3qdh"
server_key: "1af8cf18e1613538e8adc15a6e39d4c3"
server_key_iv: "a3085b0d7a75384c4b6aead68e2bf6c6"
server_key_version: "encryptionkey@0"
I20260619 14:52:36.877962 29902 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:52:36.880007 29917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:36.881031 29902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:36.881160 29902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "a58b25f56b454a3cab58087f64d48206"
format_stamp: "Formatted at 2026-06-19 14:52:36 on dist-test-slave-3qdh"
server_key: "1af8cf18e1613538e8adc15a6e39d4c3"
server_key_iv: "a3085b0d7a75384c4b6aead68e2bf6c6"
server_key_version: "encryptionkey@0"
I20260619 14:52:36.881250 29902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:36.901274 29902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:36.903920 29902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:36.904093 29902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:36.904592 29902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:36.905213 29902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:36.905272 29902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:36.905323 29902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:36.905355 29902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:36.913101 29902 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:37603
I20260619 14:52:36.913121 30030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:37603 every 8 connection(s)
I20260619 14:52:36.913801 29902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:52:36.921053 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 29902
I20260619 14:52:36.921175 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:52:36.921425 19593 external_mini_cluster.cc:1428] Setting key 30d2e532cb4b1f12c287eb704413fee9
Jun 19 14:52:36 dist-test-slave-3qdh krb5kdc[29795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880756, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:52:36.926869 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:36.915232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:35883 (local address 127.19.34.126:36295)
0619 14:52:36.915373 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:36.915378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:36.916029 (+ 651us) server_negotiation.cc:407] Connection header received
0619 14:52:36.917175 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:36.917178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:36.917218 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:36.917282 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:36.918289 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:36.918812 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:36.919749 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:36.919950 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:36.920017 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:36.922285 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:36.922288 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:36.922290 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:36.922310 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:36.924931 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:36.925417 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:36.925423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:36.925427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:36.925531 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:36.925822 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:36.925827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:36.925831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:36.926082 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:36.926176 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:36.926561 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 14:52:36.926688 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:52:36.927433 30033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:36.915532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:36295 (local address 127.19.34.65:35883)
0619 14:52:36.915904 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:36.915936 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:36.916998 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:36.917459 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:36.917465 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:36.917763 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:36.918173 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:36.918185 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:36.918899 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:36.918902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:36.919656 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:36.919664 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:36.920053 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:36.920056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:36.920186 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:36.920678 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:36.920692 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:36.922155 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:36.925032 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:36.925036 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:36.925044 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:36.925314 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:36.925618 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:36.925620 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:36.925621 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:36.925744 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:36.926184 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:36.926188 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:36.926358 (+ 170us) client_negotiation.cc:769] Sending connection context
0619 14:52:36.926485 (+ 127us) client_negotiation.cc:240] Negotiation successful
0619 14:52:36.926677 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":92,"threads_started":1}
I20260619 14:52:36.928637 30031 heartbeater.cc:347] Connected to a master server at 127.19.34.126:36295
I20260619 14:52:36.928879 30031 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:36.929412 30031 heartbeater.cc:511] Master 127.19.34.126:36295 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.34.66@KRBTEST.COM; defaulting to no policy
I20260619 14:52:36.930898 29840 ts_manager.cc:194] Registered new tserver with Master: a58b25f56b454a3cab58087f64d48206 (127.19.34.65:37603)
I20260619 14:52:36.931942 29840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:35883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
I20260619 14:52:36.945300 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:36295
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:37.044157 30036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:37.044430 30036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:37.044489 30036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:37.048352 30036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:37.048404 30036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:37.048532 30036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:52:37.052919 30036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:36295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:37.053673 30036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:37.054481 30036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:52:37.062677 30044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:37.062677 30042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:37.062808 30036 server_base.cc:1034] running on GCE node
W20260619 14:52:37.062950 30041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:37.063282 30036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:37.063814 30036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:37.064985 30036 hybrid_clock.cc:648] HybridClock initialized: now 1781880757064968 us; error 37 us; skew 500 ppm
Jun 19 14:52:37 dist-test-slave-3qdh krb5kdc[29795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880757, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:37.067373 30036 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:52:37.068249 30036 webserver.cc:460] Webserver started at http://127.19.34.66:40241/ using document root <none> and password file <none>
I20260619 14:52:37.068799 30036 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:37.068879 30036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:37.069068 30036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:37.070935 30036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "29b18305bda54429a18cddb64d5174fb"
format_stamp: "Formatted at 2026-06-19 14:52:37 on dist-test-slave-3qdh"
server_key: "313d874b35bdc54326a1b64277789d83"
server_key_iv: "230039e1e20e280aaa47aab9d457e5ee"
server_key_version: "encryptionkey@0"
I20260619 14:52:37.071457 30036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "29b18305bda54429a18cddb64d5174fb"
format_stamp: "Formatted at 2026-06-19 14:52:37 on dist-test-slave-3qdh"
server_key: "313d874b35bdc54326a1b64277789d83"
server_key_iv: "230039e1e20e280aaa47aab9d457e5ee"
server_key_version: "encryptionkey@0"
I20260619 14:52:37.074702 30036 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 14:52:37.076660 30051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:37.077721 30036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:37.077840 30036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "29b18305bda54429a18cddb64d5174fb"
format_stamp: "Formatted at 2026-06-19 14:52:37 on dist-test-slave-3qdh"
server_key: "313d874b35bdc54326a1b64277789d83"
server_key_iv: "230039e1e20e280aaa47aab9d457e5ee"
server_key_version: "encryptionkey@0"
I20260619 14:52:37.077937 30036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:37.087675 30036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:37.090340 30036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:37.090495 30036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:37.090924 30036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:37.091595 30036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:37.091660 30036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:37.091734 30036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:37.091777 30036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:37.099131 30036 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:40491
I20260619 14:52:37.099160 30164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:40491 every 8 connection(s)
I20260619 14:52:37.099793 30036 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
Jun 19 14:52:37 dist-test-slave-3qdh krb5kdc[29795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880757, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:52:37.109877 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 30036
I20260619 14:52:37.109988 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:52:37.110288 19593 external_mini_cluster.cc:1428] Setting key 1b17ad611f97ef690c8b9c685d52b7a9
I20260619 14:52:37.110523 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.101243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:55667 (local address 127.19.34.126:36295)
0619 14:52:37.101353 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:37.101356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:37.102177 (+ 821us) server_negotiation.cc:407] Connection header received
0619 14:52:37.102849 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:37.102851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:37.102891 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:37.102955 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:37.103874 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.104402 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.105379 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.105565 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.105632 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.107619 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:37.107622 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:37.107624 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:37.107644 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:37.108876 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:37.109269 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:37.109271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:37.109272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:37.109339 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:37.109576 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:37.109586 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:37.109587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:37.109724 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:37.109791 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:37.110292 (+ 501us) server_negotiation.cc:299] Negotiation successful
0619 14:52:37.110417 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:52:37.111176 30167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.101520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:36295 (local address 127.19.34.66:55667)
0619 14:52:37.102030 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:37.102063 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:37.102700 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:37.103079 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:37.103087 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:37.103363 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:37.103764 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:37.103775 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.104512 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.104514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:37.105291 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:37.105299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.105662 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.105664 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:37.105774 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.106190 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:37.106203 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:37.107465 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:37.108973 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:37.108977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:37.108986 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:37.109193 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:37.109407 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:37.109409 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:37.109410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:37.109512 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:37.109818 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:37.109824 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:37.110077 (+ 253us) client_negotiation.cc:769] Sending connection context
0619 14:52:37.110260 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 14:52:37.110529 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":95,"threads_started":1}
I20260619 14:52:37.112161 30165 heartbeater.cc:347] Connected to a master server at 127.19.34.126:36295
I20260619 14:52:37.112385 30165 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:37.112813 30165 heartbeater.cc:511] Master 127.19.34.126:36295 requested a full tablet report, sending...
I20260619 14:52:37.113812 29840 ts_manager.cc:194] Registered new tserver with Master: 29b18305bda54429a18cddb64d5174fb (127.19.34.66:40491)
I20260619 14:52:37.114322 29840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:55667
WARNING: no policy specified for kudu/127.19.34.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
I20260619 14:52:37.135370 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:36295
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:39681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:52:37.232851 30170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:52:37.233042 30170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:52:37.233093 30170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:52:37.236573 30170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:52:37.236614 30170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:52:37.236662 30170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:52:37.242161 30170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:39681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:36295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:52:37.242928 30170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:52:37.243741 30170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:52:37.252224 30170 server_base.cc:1034] running on GCE node
W20260619 14:52:37.252228 30176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:37.252269 30178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:52:37.252410 30175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:52:37.252766 30170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:52:37.253386 30170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:52:37.254547 30170 hybrid_clock.cc:648] HybridClock initialized: now 1781880757254526 us; error 38 us; skew 500 ppm
Jun 19 14:52:37 dist-test-slave-3qdh krb5kdc[29795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880757, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:52:37.256959 30170 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:52:37.257759 30170 webserver.cc:460] Webserver started at http://127.19.34.67:39547/ using document root <none> and password file <none>
I20260619 14:52:37.258246 30170 fs_manager.cc:359] Metadata directory not provided
I20260619 14:52:37.258301 30170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:52:37.258508 30170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:52:37.260499 30170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "bf0d03083a8946d5b3041faa7f4c32db"
format_stamp: "Formatted at 2026-06-19 14:52:37 on dist-test-slave-3qdh"
server_key: "ca739bc7e39b5e1f1b31b8fbb52cc0ea"
server_key_iv: "294b30c1caf7d961c3b46e8d006f81c2"
server_key_version: "encryptionkey@0"
I20260619 14:52:37.261078 30170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "bf0d03083a8946d5b3041faa7f4c32db"
format_stamp: "Formatted at 2026-06-19 14:52:37 on dist-test-slave-3qdh"
server_key: "ca739bc7e39b5e1f1b31b8fbb52cc0ea"
server_key_iv: "294b30c1caf7d961c3b46e8d006f81c2"
server_key_version: "encryptionkey@0"
I20260619 14:52:37.264225 30170 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:52:37.266172 30185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:37.267159 30170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:52:37.267261 30170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "bf0d03083a8946d5b3041faa7f4c32db"
format_stamp: "Formatted at 2026-06-19 14:52:37 on dist-test-slave-3qdh"
server_key: "ca739bc7e39b5e1f1b31b8fbb52cc0ea"
server_key_iv: "294b30c1caf7d961c3b46e8d006f81c2"
server_key_version: "encryptionkey@0"
I20260619 14:52:37.267343 30170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:52:37.291774 30170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:52:37.294476 30170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:52:37.294646 30170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:52:37.295137 30170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:52:37.295794 30170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:52:37.295833 30170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:37.295887 30170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:52:37.295918 30170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:52:37.303782 30170 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:38673
I20260619 14:52:37.303807 30298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:38673 every 8 connection(s)
I20260619 14:52:37.304593 30170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:52:37.310343 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 30170
I20260619 14:52:37.310452 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:52:37.310714 19593 external_mini_cluster.cc:1428] Setting key e059b1edc9b17435311b92d19f06eac0
Jun 19 14:52:37 dist-test-slave-3qdh krb5kdc[29795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880757, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:52:37.316722 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.306095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:48567 (local address 127.19.34.126:36295)
0619 14:52:37.306221 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:37.306226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:37.306895 (+ 669us) server_negotiation.cc:407] Connection header received
0619 14:52:37.308140 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:37.308143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:37.308189 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:37.308262 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:37.309291 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.309754 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.311164 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.311363 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.311462 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.313747 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:37.313750 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:37.313751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:37.313772 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:37.315019 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:37.315391 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:37.315394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:37.315395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:37.315461 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:37.315764 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:37.315766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:37.315767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:37.315914 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:37.315983 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:37.316523 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 14:52:37.316606 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:52:37.317394 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.306360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:36295 (local address 127.19.34.67:48567)
0619 14:52:37.306751 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:37.306788 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:37.307951 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:37.308448 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:37.308455 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:37.308784 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:52:37.309168 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:37.309179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.309840 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.309842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:37.311018 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:37.311038 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.311487 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.311491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:37.311654 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.312127 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:52:37.312144 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:52:37.313616 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:52:37.315101 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:37.315109 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:37.315118 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:37.315319 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:37.315564 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:52:37.315566 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:52:37.315567 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:52:37.315691 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:52:37.315989 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:52:37.315993 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:52:37.316172 (+ 179us) client_negotiation.cc:769] Sending connection context
0619 14:52:37.316492 (+ 320us) client_negotiation.cc:240] Negotiation successful
0619 14:52:37.316708 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":101,"threads_started":1}
I20260619 14:52:37.318398 30299 heartbeater.cc:347] Connected to a master server at 127.19.34.126:36295
I20260619 14:52:37.318614 30299 heartbeater.cc:464] Registering TS with master...
I20260619 14:52:37.319003 30299 heartbeater.cc:511] Master 127.19.34.126:36295 requested a full tablet report, sending...
I20260619 14:52:37.319847 29840 ts_manager.cc:194] Registered new tserver with Master: bf0d03083a8946d5b3041faa7f4c32db (127.19.34.67:38673)
I20260619 14:52:37.320385 29840 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:48567
I20260619 14:52:37.324060 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:52:37.331876 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.325697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52160 (local address 127.19.34.126:36295)
0619 14:52:37.325808 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:37.325812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:37.325965 (+ 153us) server_negotiation.cc:407] Connection header received
0619 14:52:37.326041 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:37.326043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:37.326074 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:37.326145 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:52:37.326750 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.327216 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.328060 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.328279 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.328400 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.329481 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:52:37.329484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:52:37.329486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:52:37.329505 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:52:37.330597 (+ 1092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:37.330949 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:37.330951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:37.330952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:37.331018 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:52:37.331221 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:52:37.331223 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:52:37.331224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:52:37.331398 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:52:37.331493 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:37.331687 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 14:52:37.331766 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:52:37.334162 29840 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:52:37.336982 29840 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:52:37.350122 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.344571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50112 (local address 127.19.34.67:38673)
0619 14:52:37.345128 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:37.345139 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:37.345161 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:52:37.345348 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:37.345356 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:37.345546 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:37.345705 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:37.346132 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.347113 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.348755 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.349460 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.349571 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.349715 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:37.349882 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 14:52:37.349976 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":78,"threads_started":1}
I20260619 14:52:37.350132 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.345180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:37603 (local address 127.0.0.1:53892)
0619 14:52:37.345740 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:37.345753 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:37.345827 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:37.346403 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:37.346406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:37.346422 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:37.346559 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:37.346565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.347530 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.347533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:37.348698 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:37.348709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.349673 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.349676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:37.349761 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.349788 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:52:37.349837 (+ 49us) client_negotiation.cc:240] Negotiation successful
0619 14:52:37.349977 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":117,"thread_start_us":76,"threads_started":1}
I20260619 14:52:37.350514 30312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.344621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:40491 (local address 127.0.0.1:43370)
0619 14:52:37.345305 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:37.345318 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:37.345393 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:37.346360 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:37.346363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:37.346380 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:37.346488 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:37.346494 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.347889 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.347892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:37.349091 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:37.349099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.350119 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.350137 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:37.350242 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.350273 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 14:52:37.350333 (+ 60us) client_negotiation.cc:240] Negotiation successful
0619 14:52:37.350388 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":81,"thread_start_us":83,"threads_started":1}
I20260619 14:52:37.350647 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.344464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:38673 (local address 127.0.0.1:50112)
0619 14:52:37.344754 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:37.344797 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:37.344898 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:37.345884 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:37.345890 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:37.345910 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:37.346015 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:37.346021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.347266 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.347269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:37.348625 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:37.348635 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.349572 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.349576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:37.349689 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.349726 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 14:52:37.349844 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 14:52:37.350429 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":6,"thread_start_us":74,"threads_started":1}
I20260619 14:52:37.350946 30314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.345317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.19.34.65:37603)
0619 14:52:37.346065 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:37.346077 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:37.346097 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:52:37.346169 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:37.346172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:37.346291 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:37.346423 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:37.346663 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.347375 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.348830 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.349525 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.349974 (+ 449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.350116 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:37.350162 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 14:52:37.350546 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":144,"threads_started":1}
I20260619 14:52:37.351114 30315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.345539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43370 (local address 127.19.34.66:40491)
0619 14:52:37.345967 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:37.345974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:37.345991 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:52:37.346044 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:37.346047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:37.346217 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:37.346620 (+ 403us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:37.346778 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.347752 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.349221 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.349995 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.350709 (+ 714us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.350854 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:37.350899 (+ 45us) server_negotiation.cc:299] Negotiation successful
0619 14:52:37.350974 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":86,"threads_started":1}
I20260619 14:52:37.353188 29965 tablet_service.cc:1467] Processing CreateTablet for tablet 279b5b37e42c46e5a87e70d560e2e388 (DEFAULT_TABLE table=test-table [id=ab2af7542a0b4fe48be06c0d04c0975d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:37.353255 30233 tablet_service.cc:1467] Processing CreateTablet for tablet 279b5b37e42c46e5a87e70d560e2e388 (DEFAULT_TABLE table=test-table [id=ab2af7542a0b4fe48be06c0d04c0975d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:37.354104 29965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 279b5b37e42c46e5a87e70d560e2e388. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:37.354192 30233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 279b5b37e42c46e5a87e70d560e2e388. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:37.354013 30099 tablet_service.cc:1467] Processing CreateTablet for tablet 279b5b37e42c46e5a87e70d560e2e388 (DEFAULT_TABLE table=test-table [id=ab2af7542a0b4fe48be06c0d04c0975d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:52:37.354837 30099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 279b5b37e42c46e5a87e70d560e2e388. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:52:37.359532 30316 tablet_bootstrap.cc:492] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206: Bootstrap starting.
I20260619 14:52:37.360575 30317 tablet_bootstrap.cc:492] T 279b5b37e42c46e5a87e70d560e2e388 P bf0d03083a8946d5b3041faa7f4c32db: Bootstrap starting.
I20260619 14:52:37.361307 30318 tablet_bootstrap.cc:492] T 279b5b37e42c46e5a87e70d560e2e388 P 29b18305bda54429a18cddb64d5174fb: Bootstrap starting.
I20260619 14:52:37.362591 30317 tablet_bootstrap.cc:654] T 279b5b37e42c46e5a87e70d560e2e388 P bf0d03083a8946d5b3041faa7f4c32db: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:37.363025 30316 tablet_bootstrap.cc:654] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:37.363059 30318 tablet_bootstrap.cc:654] T 279b5b37e42c46e5a87e70d560e2e388 P 29b18305bda54429a18cddb64d5174fb: Neither blocks nor log segments found. Creating new log.
I20260619 14:52:37.363715 30318 log.cc:824] T 279b5b37e42c46e5a87e70d560e2e388 P 29b18305bda54429a18cddb64d5174fb: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:37.363803 30317 log.cc:824] T 279b5b37e42c46e5a87e70d560e2e388 P bf0d03083a8946d5b3041faa7f4c32db: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:37.363803 30316 log.cc:824] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206: Log is configured to *not* fsync() on all Append() calls
I20260619 14:52:37.365424 30317 tablet_bootstrap.cc:492] T 279b5b37e42c46e5a87e70d560e2e388 P bf0d03083a8946d5b3041faa7f4c32db: No bootstrap required, opened a new log
I20260619 14:52:37.365573 30318 tablet_bootstrap.cc:492] T 279b5b37e42c46e5a87e70d560e2e388 P 29b18305bda54429a18cddb64d5174fb: No bootstrap required, opened a new log
I20260619 14:52:37.365573 30316 tablet_bootstrap.cc:492] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206: No bootstrap required, opened a new log
I20260619 14:52:37.365602 30317 ts_tablet_manager.cc:1397] T 279b5b37e42c46e5a87e70d560e2e388 P bf0d03083a8946d5b3041faa7f4c32db: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260619 14:52:37.365733 30316 ts_tablet_manager.cc:1397] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:52:37.365733 30318 ts_tablet_manager.cc:1397] T 279b5b37e42c46e5a87e70d560e2e388 P 29b18305bda54429a18cddb64d5174fb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:52:37.368680 30317 raft_consensus.cc:348] T 279b5b37e42c46e5a87e70d560e2e388 P bf0d03083a8946d5b3041faa7f4c32db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0d03083a8946d5b3041faa7f4c32db" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38673 } } peers { permanent_uuid: "29b18305bda54429a18cddb64d5174fb" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 40491 } } peers { permanent_uuid: "a58b25f56b454a3cab58087f64d48206" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 37603 } }
I20260619 14:52:37.368919 30317 raft_consensus.cc:374] T 279b5b37e42c46e5a87e70d560e2e388 P bf0d03083a8946d5b3041faa7f4c32db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:37.368974 30317 raft_consensus.cc:729] T 279b5b37e42c46e5a87e70d560e2e388 P bf0d03083a8946d5b3041faa7f4c32db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf0d03083a8946d5b3041faa7f4c32db, State: Initialized, Role: FOLLOWER
I20260619 14:52:37.369370 30317 consensus_queue.cc:260] T 279b5b37e42c46e5a87e70d560e2e388 P bf0d03083a8946d5b3041faa7f4c32db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0d03083a8946d5b3041faa7f4c32db" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38673 } } peers { permanent_uuid: "29b18305bda54429a18cddb64d5174fb" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 40491 } } peers { permanent_uuid: "a58b25f56b454a3cab58087f64d48206" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 37603 } }
I20260619 14:52:37.369539 30316 raft_consensus.cc:348] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0d03083a8946d5b3041faa7f4c32db" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38673 } } peers { permanent_uuid: "29b18305bda54429a18cddb64d5174fb" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 40491 } } peers { permanent_uuid: "a58b25f56b454a3cab58087f64d48206" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 37603 } }
I20260619 14:52:37.369534 30318 raft_consensus.cc:348] T 279b5b37e42c46e5a87e70d560e2e388 P 29b18305bda54429a18cddb64d5174fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0d03083a8946d5b3041faa7f4c32db" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38673 } } peers { permanent_uuid: "29b18305bda54429a18cddb64d5174fb" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 40491 } } peers { permanent_uuid: "a58b25f56b454a3cab58087f64d48206" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 37603 } }
I20260619 14:52:37.369768 30316 raft_consensus.cc:374] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:37.369767 30318 raft_consensus.cc:374] T 279b5b37e42c46e5a87e70d560e2e388 P 29b18305bda54429a18cddb64d5174fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:52:37.369834 30316 raft_consensus.cc:729] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a58b25f56b454a3cab58087f64d48206, State: Initialized, Role: FOLLOWER
I20260619 14:52:37.369827 30318 raft_consensus.cc:729] T 279b5b37e42c46e5a87e70d560e2e388 P 29b18305bda54429a18cddb64d5174fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b18305bda54429a18cddb64d5174fb, State: Initialized, Role: FOLLOWER
I20260619 14:52:37.370136 30299 heartbeater.cc:503] Master 127.19.34.126:36295 was elected leader, sending a full tablet report...
I20260619 14:52:37.370275 30317 ts_tablet_manager.cc:1428] T 279b5b37e42c46e5a87e70d560e2e388 P bf0d03083a8946d5b3041faa7f4c32db: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:52:37.370257 30318 consensus_queue.cc:260] T 279b5b37e42c46e5a87e70d560e2e388 P 29b18305bda54429a18cddb64d5174fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0d03083a8946d5b3041faa7f4c32db" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38673 } } peers { permanent_uuid: "29b18305bda54429a18cddb64d5174fb" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 40491 } } peers { permanent_uuid: "a58b25f56b454a3cab58087f64d48206" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 37603 } }
I20260619 14:52:37.370257 30316 consensus_queue.cc:260] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0d03083a8946d5b3041faa7f4c32db" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38673 } } peers { permanent_uuid: "29b18305bda54429a18cddb64d5174fb" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 40491 } } peers { permanent_uuid: "a58b25f56b454a3cab58087f64d48206" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 37603 } }
I20260619 14:52:37.370934 30165 heartbeater.cc:503] Master 127.19.34.126:36295 was elected leader, sending a full tablet report...
I20260619 14:52:37.371134 30316 ts_tablet_manager.cc:1428] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260619 14:52:37.371146 30318 ts_tablet_manager.cc:1428] T 279b5b37e42c46e5a87e70d560e2e388 P 29b18305bda54429a18cddb64d5174fb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:52:37.371434 30031 heartbeater.cc:503] Master 127.19.34.126:36295 was elected leader, sending a full tablet report...
I20260619 14:52:37.405222 30323 raft_consensus.cc:482] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:52:37.405376 30323 raft_consensus.cc:504] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0d03083a8946d5b3041faa7f4c32db" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38673 } } peers { permanent_uuid: "29b18305bda54429a18cddb64d5174fb" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 40491 } } peers { permanent_uuid: "a58b25f56b454a3cab58087f64d48206" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 37603 } }
I20260619 14:52:37.406347 30323 leader_election.cc:288] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf0d03083a8946d5b3041faa7f4c32db (127.19.34.67:38673), 29b18305bda54429a18cddb64d5174fb (127.19.34.66:40491)
I20260619 14:52:37.409950 30325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.406548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:40491 (local address 127.19.34.65:37001)
0619 14:52:37.406849 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:37.406875 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:37.406973 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:37.407228 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:37.407231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:37.407244 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:37.407360 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:37.407372 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.408072 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.408074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:37.408921 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:37.408930 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.409555 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.409558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:37.409644 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.409679 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 14:52:37.409742 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 14:52:37.409789 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":2,"thread_start_us":59,"threads_started":1}
I20260619 14:52:37.409965 30315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.406689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:37001 (local address 127.19.34.66:40491)
0619 14:52:37.406812 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:37.406816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:37.406955 (+ 139us) server_negotiation.cc:407] Connection header received
0619 14:52:37.407089 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:37.407092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:37.407137 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:37.407210 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:37.407473 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.407963 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.409060 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.409448 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.409556 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.409617 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:37.409776 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 14:52:37.409851 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:52:37.409999 30033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.406523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:38673 (local address 127.19.34.65:40719)
0619 14:52:37.406701 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:37.406716 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:52:37.406809 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:52:37.407081 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:52:37.407084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:52:37.407102 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:52:37.407238 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:37.407244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.408008 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.408012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:37.408914 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:52:37.408923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.409599 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.409602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:52:37.409716 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.409754 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 14:52:37.409830 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:52:37.409892 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":71}
I20260619 14:52:37.410056 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.406582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:40719 (local address 127.19.34.67:38673)
0619 14:52:37.406702 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:37.406707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:37.406789 (+ 82us) server_negotiation.cc:407] Connection header received
0619 14:52:37.406922 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:37.406927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:37.406984 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:37.407059 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:52:37.407345 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.407899 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.409028 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.409468 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.409590 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.409649 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:37.409855 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 14:52:37.409920 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:52:37.410465 30119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "279b5b37e42c46e5a87e70d560e2e388" candidate_uuid: "a58b25f56b454a3cab58087f64d48206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b18305bda54429a18cddb64d5174fb" is_pre_election: true
I20260619 14:52:37.410543 30253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "279b5b37e42c46e5a87e70d560e2e388" candidate_uuid: "a58b25f56b454a3cab58087f64d48206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf0d03083a8946d5b3041faa7f4c32db" is_pre_election: true
I20260619 14:52:37.410794 30119 raft_consensus.cc:2454] T 279b5b37e42c46e5a87e70d560e2e388 P 29b18305bda54429a18cddb64d5174fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a58b25f56b454a3cab58087f64d48206 in term 0.
I20260619 14:52:37.410815 30253 raft_consensus.cc:2454] T 279b5b37e42c46e5a87e70d560e2e388 P bf0d03083a8946d5b3041faa7f4c32db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a58b25f56b454a3cab58087f64d48206 in term 0.
I20260619 14:52:37.411198 29920 leader_election.cc:302] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b18305bda54429a18cddb64d5174fb, a58b25f56b454a3cab58087f64d48206; no voters:
I20260619 14:52:37.411434 30323 raft_consensus.cc:2781] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:52:37.411535 30323 raft_consensus.cc:482] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:52:37.411576 30323 raft_consensus.cc:3037] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:37.412597 30323 raft_consensus.cc:504] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0d03083a8946d5b3041faa7f4c32db" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38673 } } peers { permanent_uuid: "29b18305bda54429a18cddb64d5174fb" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 40491 } } peers { permanent_uuid: "a58b25f56b454a3cab58087f64d48206" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 37603 } }
I20260619 14:52:37.412891 30323 leader_election.cc:288] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206 [CANDIDATE]: Term 1 election: Requested vote from peers bf0d03083a8946d5b3041faa7f4c32db (127.19.34.67:38673), 29b18305bda54429a18cddb64d5174fb (127.19.34.66:40491)
I20260619 14:52:37.413163 30253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "279b5b37e42c46e5a87e70d560e2e388" candidate_uuid: "a58b25f56b454a3cab58087f64d48206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf0d03083a8946d5b3041faa7f4c32db"
I20260619 14:52:37.413215 30119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "279b5b37e42c46e5a87e70d560e2e388" candidate_uuid: "a58b25f56b454a3cab58087f64d48206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29b18305bda54429a18cddb64d5174fb"
I20260619 14:52:37.413275 30253 raft_consensus.cc:3037] T 279b5b37e42c46e5a87e70d560e2e388 P bf0d03083a8946d5b3041faa7f4c32db [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:37.413306 30119 raft_consensus.cc:3037] T 279b5b37e42c46e5a87e70d560e2e388 P 29b18305bda54429a18cddb64d5174fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:52:37.414465 30119 raft_consensus.cc:2454] T 279b5b37e42c46e5a87e70d560e2e388 P 29b18305bda54429a18cddb64d5174fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a58b25f56b454a3cab58087f64d48206 in term 1.
I20260619 14:52:37.414465 30253 raft_consensus.cc:2454] T 279b5b37e42c46e5a87e70d560e2e388 P bf0d03083a8946d5b3041faa7f4c32db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a58b25f56b454a3cab58087f64d48206 in term 1.
I20260619 14:52:37.414749 29918 leader_election.cc:302] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a58b25f56b454a3cab58087f64d48206, bf0d03083a8946d5b3041faa7f4c32db; no voters:
W20260619 14:52:37.414813 30032 tablet.cc:2369] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:52:37.414917 30323 raft_consensus.cc:2781] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:52:37.415125 30323 raft_consensus.cc:686] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206 [term 1 LEADER]: Becoming Leader. State: Replica: a58b25f56b454a3cab58087f64d48206, State: Running, Role: LEADER
I20260619 14:52:37.415408 30323 consensus_queue.cc:237] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0d03083a8946d5b3041faa7f4c32db" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38673 } } peers { permanent_uuid: "29b18305bda54429a18cddb64d5174fb" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 40491 } } peers { permanent_uuid: "a58b25f56b454a3cab58087f64d48206" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 37603 } }
I20260619 14:52:37.418181 29839 catalog_manager.cc:5495] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206 reported cstate change: term changed from 0 to 1, leader changed from <none> to a58b25f56b454a3cab58087f64d48206 (127.19.34.65). New cstate: current_term: 1 leader_uuid: "a58b25f56b454a3cab58087f64d48206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0d03083a8946d5b3041faa7f4c32db" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29b18305bda54429a18cddb64d5174fb" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 40491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a58b25f56b454a3cab58087f64d48206" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 37603 } health_report { overall_health: HEALTHY } } }
I20260619 14:52:37.445137 30314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.442156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53902 (local address 127.19.34.65:37603)
0619 14:52:37.442264 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 14:52:37.442267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:52:37.442279 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:52:37.442383 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:52:37.442385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:52:37.442444 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:52:37.442539 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:52:37.442736 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.443179 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.443923 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:52:37.444100 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:52:37.444197 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:52:37.444488 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:52:37.444850 (+ 362us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:52:37.444935 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:52:37.444984 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 14:52:37.445026 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:52:37.450157 30119 raft_consensus.cc:1261] T 279b5b37e42c46e5a87e70d560e2e388 P 29b18305bda54429a18cddb64d5174fb [term 1 FOLLOWER]: Refusing update from remote peer a58b25f56b454a3cab58087f64d48206: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:37.450166 30253 raft_consensus.cc:1261] T 279b5b37e42c46e5a87e70d560e2e388 P bf0d03083a8946d5b3041faa7f4c32db [term 1 FOLLOWER]: Refusing update from remote peer a58b25f56b454a3cab58087f64d48206: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:52:37.450783 30326 consensus_queue.cc:1035] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf0d03083a8946d5b3041faa7f4c32db" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:37.450954 30323 consensus_queue.cc:1035] T 279b5b37e42c46e5a87e70d560e2e388 P a58b25f56b454a3cab58087f64d48206 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29b18305bda54429a18cddb64d5174fb" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 40491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:52:37.457664 30330 mvcc.cc:205] Tried to move back new op lower bound from 7298583582509993984 to 7298583582375706624. Current Snapshot: MvccSnapshot[applied={T|T < 7298583582509993984}]
I20260619 14:52:37.464552 29839 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:37.464676 29839 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:52:37.466746 29839 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e15b567e968c48cb9e17accc7da25724: Sending DeleteTablet for 3 replicas of tablet 279b5b37e42c46e5a87e70d560e2e388
I20260619 14:52:37.467360 30233 tablet_service.cc:1514] Processing DeleteTablet for tablet 279b5b37e42c46e5a87e70d560e2e388 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:37 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:50112
I20260619 14:52:37.467667 30099 tablet_service.cc:1514] Processing DeleteTablet for tablet 279b5b37e42c46e5a87e70d560e2e388 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:37 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:43370
I20260619 14:52:37.467847 30339 tablet_replica.cc:330] T 279b5b37e42c46e5a87e70d560e2e388 P bf0d03083a8946d5b3041faa7f4c32db: stopping tablet replica
I20260619 14:52:37.468075 30340 tablet_replica.cc:330] T 279b5b37e42c46e5a87e70d560e2e388 P 29b18305bda54429a18cddb64d5174fb: stopping tablet replica
I20260619 14:52:37.468083 29965 tablet_service.cc:1514] Processing DeleteTablet for tablet 279b5b37e42c46e5a87e70d560e2e388 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:52:37 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:53892
I20260619 14:52:37.468178 30339 raft_consensus.cc:2229] T 279b5b37e42c46e5a87e70d560e2e388 P bf0d03083a8946d5b3041faa7f4c32db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:37.468321 30340 raft_consensus.cc:2229] T 279b5b37e42c46e5a87e70d560e2e388 P 29b18305bda54429a18cddb64d5174fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:52:37.468461 30339 raft_consensus.cc:2258] T 279b5b37e42c46e5a87e70d560e2e388 P bf0d03083a8946d5b3041faa7f4c32db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:37.468523 30340 raft_consensus.cc:2258] T 279b5b37e42c46e5a87e70d560e2e388 P 29b18305bda54429a18cddb64d5174fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:52:37.468598 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 29902
I20260619 14:52:37.471105 30339 ts_tablet_manager.cc:1905] T 279b5b37e42c46e5a87e70d560e2e388 P bf0d03083a8946d5b3041faa7f4c32db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:52:37.473238 30339 ts_tablet_manager.cc:1918] T 279b5b37e42c46e5a87e70d560e2e388 P bf0d03083a8946d5b3041faa7f4c32db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:52:37.473315 30339 log.cc:1196] T 279b5b37e42c46e5a87e70d560e2e388 P bf0d03083a8946d5b3041faa7f4c32db: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/279b5b37e42c46e5a87e70d560e2e388
I20260619 14:52:37.473623 30339 ts_tablet_manager.cc:1939] T 279b5b37e42c46e5a87e70d560e2e388 P bf0d03083a8946d5b3041faa7f4c32db: Deleting consensus metadata
I20260619 14:52:37.474654 29824 catalog_manager.cc:4841] TS bf0d03083a8946d5b3041faa7f4c32db (127.19.34.67:38673): tablet 279b5b37e42c46e5a87e70d560e2e388 (table test-table [id=ab2af7542a0b4fe48be06c0d04c0975d]) successfully deleted
W20260619 14:52:37.474836 29826 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.34.65:37603 (error 108)
I20260619 14:52:37.475574 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 30036
I20260619 14:52:37.475814 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.475495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:37603 (local address 127.0.0.1:53912)
0619 14:52:37.475643 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:37.475720 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.65:37603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 14:52:37.476333 29826 catalog_manager.cc:4584] TS a58b25f56b454a3cab58087f64d48206 (127.19.34.65:37603): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 279b5b37e42c46e5a87e70d560e2e388: Network error: Client connection negotiation failed: client connection to 127.19.34.65:37603: connect: Connection refused (error 111)
I20260619 14:52:37.481238 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 30170
I20260619 14:52:37.481539 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:52:37.481284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:40491 (local address 127.0.0.1:43378)
0619 14:52:37.481405 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 14:52:37.481460 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.66:40491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 14:52:37.486169 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 29808
2026-06-19T14:52:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2097 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:52:37 dist-test-slave-3qdh krb5kdc[30345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:52:37 dist-test-slave-3qdh krb5kdc[30345](info): set up 2 sockets
Jun 19 14:52:37 dist-test-slave-3qdh krb5kdc[30345](info): commencing operation
krb5kdc: starting...
W20260619 14:52:38.448968 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:52:38 dist-test-slave-3qdh krb5kdc[30345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:52:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:52:38Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.34.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.34.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.34.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/rangeradmin_127.19.34.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.34.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/rangeradmin_127.19.34.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.34.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.34.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.34.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/rangerlookup_127.19.34.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.34.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/rangerlookup_127.19.34.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.34.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.34.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.34.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/HTTP_127.19.34.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.34.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/HTTP_127.19.34.84@KRBTEST.COM.keytab.
I20260619 14:52:38.584501 19593 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0Qcekt/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/postgres -l logfile start
2026-06-19 14:52:40.730 UTC [30375] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 14:52:40.730 UTC [30375] LOG: listening on IPv4 address "127.19.34.84", port 50739
2026-06-19 14:52:40.733 UTC [30375] LOG: listening on Unix socket "/tmp/.s.PGSQL.50739"
2026-06-19 14:52:40.753 UTC [30378] LOG: database system was shut down at 2026-06-19 14:52:39 UTC
2026-06-19 14:52:40.755 UTC [30375] LOG: database system is ready to accept connections
I20260619 14:52:41.636644 19593 mini_postgres.cc:96] Postgres bound to 50739
2026-06-19 14:52:41.642 UTC [30386] FATAL: database "slave" does not exist
127.19.34.84:50739 - accepting connections
I20260619 14:52:41.642952 19593 mini_ranger.cc:161] Starting Ranger...
I20260619 14:52:41.656798 19593 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 14:52:42.061841 19593 mini_ranger.cc:87] Created ranger Postgres database
I20260619 14:52:42.061950 19593 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ranger-admin
2026-06-19 14:52:42,321 [I] DB FLAVOR :POSTGRES
2026-06-19 14:52:42,321 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:52:42,321 [I] Checking connection..
2026-06-19 14:52:42,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 14:52:42,522 [I] Checking connection passed.
2026-06-19 14:52:42,522 [I] --------- Verifying version history table ---------
2026-06-19 14:52:42,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:52:42,751 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:52:42,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:52:42,991 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:52:42,992 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 14:52:42,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 14:52:43.219 UTC [30450] WARNING: there is no transaction in progress
2026-06-19 14:52:43,225 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 14:52:43,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:52:43,466 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 14:52:43,467 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:52:43,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 14:52:43,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 14:52:43,933 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3qdh.c.gcp-upstream.internal','N') ;"
2026-06-19 14:52:44,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:52:44,363 [I] Table x_portal_user does not exist in database ranger
2026-06-19 14:52:44,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:52:44,585 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 14:52:44,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:52:44,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:52:45,049 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 14:52:45,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 14:52:45.899 UTC [30558] WARNING: there is no transaction in progress
2026-06-19 14:52:45.913 UTC [30558] WARNING: there is no transaction in progress
2026-06-19 14:52:45.928 UTC [30558] WARNING: there is no transaction in progress
2026-06-19 14:52:45.941 UTC [30558] WARNING: there is no transaction in progress
2026-06-19 14:52:45.955 UTC [30558] WARNING: there is no transaction in progress
2026-06-19 14:52:45.970 UTC [30558] WARNING: there is no transaction in progress
2026-06-19 14:52:46.017 UTC [30558] WARNING: there is no transaction in progress
2026-06-19 14:52:46.023 UTC [30558] WARNING: there is no transaction in progress
2026-06-19 14:52:46.036 UTC [30558] WARNING: there is no transaction in progress
2026-06-19 14:52:46.044 UTC [30558] WARNING: there is no transaction in progress
2026-06-19 14:52:46.053 UTC [30558] WARNING: there is no transaction in progress
2026-06-19 14:52:46.066 UTC [30558] WARNING: there is no transaction in progress
2026-06-19 14:52:46.074 UTC [30558] WARNING: there is no transaction in progress
2026-06-19 14:52:46.080 UTC [30558] WARNING: there is no transaction in progress
2026-06-19 14:52:46.085 UTC [30558] WARNING: there is no transaction in progress
2026-06-19 14:52:46.091 UTC [30558] WARNING: there is no transaction in progress
2026-06-19 14:52:46,609 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 14:52:46,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 14:52:46,816 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 14:52:46,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:52:47,041 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 14:52:47,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:52:47,270 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 14:52:47,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:52:47,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:52:47,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3qdh.c.gcp-upstream.internal';"
2026-06-19 14:52:47,891 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 14:52:47,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 14:52:48,097 [I] DB_PATCHES have already been applied
I20260619 14:52:48.102557 19593 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ranger-admin:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0Qcekt/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 14:52:48.102617 19593 mini_ranger.cc:193] Using host: 127.19.34.84
I20260619 14:52:48.105688 19593 mini_ranger.cc:227] Ranger admin URL: http://127.19.34.84:53227
Jun 19, 2026 2:52:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:52:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:52:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:52:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:52:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:52:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:52:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 14:52:49.146190 19593 mini_ranger.cc:160] Time spent starting Ranger: real 7.503s user 0.005s sys 0.004s
Jun 19, 2026 2:53:07 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:53:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 2:53:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:53:07 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:53:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:53:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:53:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:53:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:53:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:53:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:53:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:53:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:53:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:53:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:53:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:53:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:53:07 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:53:08 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 14:53:09.138270 19593 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.34.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.34.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.34.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/rangerkms_127.19.34.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.34.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/rangerkms_127.19.34.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.34.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.34.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.34.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/HTTP_127.19.34.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.34.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/HTTP_127.19.34.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 14:53:09 dist-test-slave-3qdh krb5kdc[30345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880789, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 14:53:09.219825 19593 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 14:53:09.228434 19593 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 14:53:09.994263 19593 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 14:53:09.994340 19593 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ranger-kms
I20260619 14:53:09.994428 19593 mini_ranger_kms.cc:227] Using postgres at 127.19.34.84:50739
2026-06-19 14:53:10,157 [I] DB FLAVOR :POSTGRES
2026-06-19 14:53:10,158 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:53:10,158 [I] Checking connection
2026-06-19 14:53:10,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 14:53:10,422 [I] connection success
2026-06-19 14:53:10,423 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 14:53:10,423 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 14:53:10,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 14:53:10,635 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 14:53:10,635 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:53:10,635 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 14:53:10,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:50739/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 14:53:10,874 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 14:53:11.295162 19593 mini_ranger_kms.cc:310] Created kms service
I20260619 14:53:11.488035 19593 mini_ranger_kms.cc:326] Created kudu user
I20260619 14:53:11.527144 19593 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 14:53:11 dist-test-slave-3qdh krb5kdc[30345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880789, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
I20260619 14:53:11.887854 19593 mini_ranger_kms.cc:384] Added ranger policy
I20260619 14:53:11.888041 19593 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ranger-kms:/tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 14:53:11.888068 19593 mini_ranger_kms.cc:240] Using host: 127.19.34.84
I20260619 14:53:11.891340 19593 mini_ranger_kms.cc:276] Ranger KMS PID: 30738
I20260619 14:53:11.891474 19593 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.34.84:59307
Jun 19, 2026 2:53:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:53:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:53:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:53:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:53:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:53:13 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 14:53:14.047709 19593 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.828s user 0.023s sys 0.124s
I20260619 14:53:14.047855 19593 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 14:53:14.047950 19593 mini_ranger_kms.cc:401] 127.19.34.84:59307/kms/v1/keys
Jun 19 14:53:14 dist-test-slave-3qdh krb5kdc[30345](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781880794, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.34.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 14:53:14 dist-test-slave-3qdh krb5kdc[30345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880794, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.34.84@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
Jun 19 14:53:14 dist-test-slave-3qdh krb5kdc[30345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880794, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.34.84@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
Jun 19 14:53:16 dist-test-slave-3qdh krb5kdc[30345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:53:17.001174 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:45541
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:45521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:45541
--ranger_config_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.34.84:59307/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:53:17.098814 30789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:53:17.098996 30789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:53:17.099026 30789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:53:17.102455 30789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:53:17.102494 30789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:53:17.102509 30789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:53:17.102522 30789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:53:17.102535 30789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:53:17.106861 30789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:45521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.34.84:59307/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:45541
--ranger_config_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:45541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:53:17.107522 30789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:53:17.108330 30789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:53:17.115480 30789 server_base.cc:1034] running on GCE node
W20260619 14:53:17.115458 30797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:17.115430 30795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:17.115440 30794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:17.116137 30789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:53:17.117076 30789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:53:17.118251 30789 hybrid_clock.cc:648] HybridClock initialized: now 1781880797118238 us; error 27 us; skew 500 ppm
Jun 19 14:53:17 dist-test-slave-3qdh krb5kdc[30345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880797, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:53:17.125051 30789 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:53:17.125921 30789 webserver.cc:460] Webserver started at http://127.19.34.126:45741/ using document root <none> and password file <none>
I20260619 14:53:17.126379 30789 fs_manager.cc:359] Metadata directory not provided
I20260619 14:53:17.126453 30789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:53:17.126660 30789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:53:17 dist-test-slave-3qdh krb5kdc[30345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880797, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
I20260619 14:53:17.170660 30789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "49689aef3a874060b1d1a817f357b7ce"
format_stamp: "Formatted at 2026-06-19 14:53:17 on dist-test-slave-3qdh"
server_key: "fcfdac1d10c359d44f01debe2fb038af"
server_key_iv: "d744000f4f22cfca6f7bf8587345414c"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:17.171243 30789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "49689aef3a874060b1d1a817f357b7ce"
format_stamp: "Formatted at 2026-06-19 14:53:17 on dist-test-slave-3qdh"
server_key: "fcfdac1d10c359d44f01debe2fb038af"
server_key_iv: "d744000f4f22cfca6f7bf8587345414c"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:17.174564 30789 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 14:53:17.187661 30804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:17.189174 30789 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 14:53:17.189303 30789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "49689aef3a874060b1d1a817f357b7ce"
format_stamp: "Formatted at 2026-06-19 14:53:17 on dist-test-slave-3qdh"
server_key: "fcfdac1d10c359d44f01debe2fb038af"
server_key_iv: "d744000f4f22cfca6f7bf8587345414c"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:17.189404 30789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:53:17.199810 30789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:53:17.202453 30789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:53:17.202615 30789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:53:17.210098 30789 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:45541
I20260619 14:53:17.210160 30856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:45541 every 8 connection(s)
I20260619 14:53:17.210912 30789 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:53:17.213958 30857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:53:17.216689 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 30789
I20260619 14:53:17.216806 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:53:17.217072 19593 external_mini_cluster.cc:1428] Setting key d6d786373ae973fe652bf494059a1285
I20260619 14:53:17.221065 30857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce: Bootstrap starting.
I20260619 14:53:17.223456 30857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce: Neither blocks nor log segments found. Creating new log.
I20260619 14:53:17.224252 30857 log.cc:824] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:53:17 dist-test-slave-3qdh krb5kdc[30345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:53:17.226235 30857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce: No bootstrap required, opened a new log
I20260619 14:53:17.229431 30860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:17.218232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49686 (local address 127.19.34.126:45541)
0619 14:53:17.218580 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:17.218592 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:17.218622 (+ 30us) server_negotiation.cc:407] Connection header received
0619 14:53:17.219138 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:17.219151 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:17.219406 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:17.219679 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:17.220421 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.221366 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.222196 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:17.222578 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:17.222721 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:17.225077 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:17.225082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:17.225092 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:17.225115 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:17.227056 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:17.227451 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:17.227460 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:17.227464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:17.227547 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:17.227832 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:17.227836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:17.227837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:17.228154 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:17.228357 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:17.228608 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 14:53:17.228869 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":109,"threads_started":1}
I20260619 14:53:17.230587 30857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49689aef3a874060b1d1a817f357b7ce" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45541 } }
I20260619 14:53:17.230819 30857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:53:17.230929 30857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49689aef3a874060b1d1a817f357b7ce, State: Initialized, Role: FOLLOWER
I20260619 14:53:17.231383 30857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49689aef3a874060b1d1a817f357b7ce" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45541 } }
I20260619 14:53:17.231628 30857 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:53:17.231709 30857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:53:17.231791 30857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:53:17.232762 30857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49689aef3a874060b1d1a817f357b7ce" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45541 } }
I20260619 14:53:17.233059 30857 leader_election.cc:302] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49689aef3a874060b1d1a817f357b7ce; no voters:
I20260619 14:53:17.233367 30857 leader_election.cc:288] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:53:17.233455 30862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:53:17.233671 30862 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce [term 1 LEADER]: Becoming Leader. State: Replica: 49689aef3a874060b1d1a817f357b7ce, State: Running, Role: LEADER
I20260619 14:53:17.233969 30862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49689aef3a874060b1d1a817f357b7ce" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45541 } }
I20260619 14:53:17.234421 30857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:53:17.234925 30862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49689aef3a874060b1d1a817f357b7ce. Latest consensus state: current_term: 1 leader_uuid: "49689aef3a874060b1d1a817f357b7ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49689aef3a874060b1d1a817f357b7ce" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45541 } } }
I20260619 14:53:17.235368 30862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce [sys.catalog]: This master's current role is: LEADER
I20260619 14:53:17.235263 30864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49689aef3a874060b1d1a817f357b7ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49689aef3a874060b1d1a817f357b7ce" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45541 } } }
I20260619 14:53:17.235481 30864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce [sys.catalog]: This master's current role is: LEADER
I20260619 14:53:17.236435 30871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:53:17.236744 30857 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 14:53:17.241906 30871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:53:17.248966 30871 catalog_manager.cc:1347] Generated new cluster ID: 1ad87c35d910488282120eeb2008d734
I20260619 14:53:17.249014 30871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:53:17.278025 30871 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:53:17.279071 30871 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:53:17.284370 30871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce: Generated new TSK 0
I20260619 14:53:17.285161 30871 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.34.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
I20260619 14:53:17.887650 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.34.84:59307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:45541
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:45521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:53:18.004719 30902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:53:18.004904 30902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:53:18.004935 30902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:53:18.008514 30902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:53:18.008575 30902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:53:18.008639 30902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:53:18.014065 30902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:45521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.34.84:59307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:45541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:53:18.014930 30902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:53:18.016016 30902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:53:18.026036 30910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:18.026036 30909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:18.026036 30912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:18.026302 30902 server_base.cc:1034] running on GCE node
I20260619 14:53:18.026633 30902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:53:18.027148 30902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:53:18.028353 30902 hybrid_clock.cc:648] HybridClock initialized: now 1781880798028319 us; error 49 us; skew 500 ppm
Jun 19 14:53:18 dist-test-slave-3qdh krb5kdc[30345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880798, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:53:18.031301 30902 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:53:18.032202 30902 webserver.cc:460] Webserver started at http://127.19.34.65:44153/ using document root <none> and password file <none>
I20260619 14:53:18.032725 30902 fs_manager.cc:359] Metadata directory not provided
I20260619 14:53:18.032807 30902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:53:18.033021 30902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:53:18 dist-test-slave-3qdh krb5kdc[30345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880798, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
I20260619 14:53:18.056749 30902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "edaefb4f27c94f3585b9314461f8f855"
format_stamp: "Formatted at 2026-06-19 14:53:18 on dist-test-slave-3qdh"
server_key: "9e5f2720e8aa454d87800b0e54780732"
server_key_iv: "a5092661495ab6368f77e9ea5516dfe1"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:18.057438 30902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "edaefb4f27c94f3585b9314461f8f855"
format_stamp: "Formatted at 2026-06-19 14:53:18 on dist-test-slave-3qdh"
server_key: "9e5f2720e8aa454d87800b0e54780732"
server_key_iv: "a5092661495ab6368f77e9ea5516dfe1"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:18.061821 30902 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:53:18.075270 30919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:18.076695 30902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:53:18.076808 30902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "edaefb4f27c94f3585b9314461f8f855"
format_stamp: "Formatted at 2026-06-19 14:53:18 on dist-test-slave-3qdh"
server_key: "9e5f2720e8aa454d87800b0e54780732"
server_key_iv: "a5092661495ab6368f77e9ea5516dfe1"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:18.076905 30902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:53:18.100129 30902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:53:18.103197 30902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:53:18.103380 30902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:53:18.103972 30902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:53:18.104951 30902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:53:18.105016 30902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:18.105089 30902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:53:18.105118 30902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:18.122475 30902 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:41139
I20260619 14:53:18.123409 30902 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:53:18.124747 31034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:41139 every 8 connection(s)
I20260619 14:53:18.133230 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 30902
I20260619 14:53:18.133347 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:53:18.133637 19593 external_mini_cluster.cc:1428] Setting key b4750d0ac2806f67adaa21247e522d18
Jun 19 14:53:18 dist-test-slave-3qdh krb5kdc[30345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880798, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:53:18.139568 31038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:18.126701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45541 (local address 127.19.34.65:46189)
0619 14:53:18.128167 (+ 1466us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:18.128209 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:18.129054 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:18.129591 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:18.129598 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:18.129812 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:53:18.130204 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:18.130215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.131009 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.131014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:18.131847 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:18.131855 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.132396 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.132400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:18.132561 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:18.133053 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:53:18.133070 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:53:18.134693 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:53:18.136739 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:18.136746 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:18.136759 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:18.137066 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:18.137360 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:18.137365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:18.137367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:18.137512 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:18.137952 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:53:18.137960 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:53:18.138215 (+ 255us) client_negotiation.cc:769] Sending connection context
0619 14:53:18.138378 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 14:53:18.138636 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1286,"thread_start_us":109,"threads_started":1}
I20260619 14:53:18.140331 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:18.126295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:46189 (local address 127.19.34.126:45541)
0619 14:53:18.129245 (+ 2950us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:18.129252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:18.129267 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:53:18.129307 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:18.129310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:18.129359 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:18.129453 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:18.130347 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.130873 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.131972 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.132246 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.132371 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:18.134851 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:18.134854 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:18.134856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:18.134876 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:18.136607 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:18.137179 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:18.137183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:18.137186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:18.137271 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:18.137607 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:18.137610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:18.137612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:18.137820 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:18.137902 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:18.139975 (+ 2073us) server_negotiation.cc:299] Negotiation successful
0619 14:53:18.140105 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2871,"thread_start_us":88,"threads_started":1}
I20260619 14:53:18.141993 31035 heartbeater.cc:347] Connected to a master server at 127.19.34.126:45541
I20260619 14:53:18.142243 31035 heartbeater.cc:464] Registering TS with master...
I20260619 14:53:18.142761 31035 heartbeater.cc:511] Master 127.19.34.126:45541 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.34.66@KRBTEST.COM; defaulting to no policy
I20260619 14:53:18.144433 30821 ts_manager.cc:194] Registered new tserver with Master: edaefb4f27c94f3585b9314461f8f855 (127.19.34.65:41139)
I20260619 14:53:18.145586 30821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:46189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
I20260619 14:53:18.159881 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.34.84:59307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:45541
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:45521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:53:18 dist-test-slave-3qdh krb5kdc[30345](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781880798, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 14:53:18.283488 31041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:53:18.283762 31041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:53:18.283821 31041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:53:18.289199 31041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:53:18.289312 31041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:53:18.289434 31041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:53:18.296397 31041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:45521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.34.84:59307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:45541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:53:18.297600 31041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:53:18.298755 31041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:53:18.310252 31050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:18.310359 31048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:18.310655 31047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:18.311484 31041 server_base.cc:1034] running on GCE node
I20260619 14:53:18.311906 31041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:53:18.312639 31041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:53:18.315935 31041 hybrid_clock.cc:648] HybridClock initialized: now 1781880798315919 us; error 36 us; skew 500 ppm
Jun 19 14:53:18 dist-test-slave-3qdh krb5kdc[30345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880798, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:53:18.319360 31041 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:53:18.320282 31041 webserver.cc:460] Webserver started at http://127.19.34.66:40101/ using document root <none> and password file <none>
I20260619 14:53:18.321103 31041 fs_manager.cc:359] Metadata directory not provided
I20260619 14:53:18.321250 31041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:53:18.321532 31041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:53:18 dist-test-slave-3qdh krb5kdc[30345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880798, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
I20260619 14:53:18.337324 31041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "031be3ff9dd144b49fc3c01641a1cd48"
format_stamp: "Formatted at 2026-06-19 14:53:18 on dist-test-slave-3qdh"
server_key: "6213d016a670be737c6ed0d80365c2d5"
server_key_iv: "a78bfc64b156352f7804cff7c2a69f5e"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:18.338855 31041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "031be3ff9dd144b49fc3c01641a1cd48"
format_stamp: "Formatted at 2026-06-19 14:53:18 on dist-test-slave-3qdh"
server_key: "6213d016a670be737c6ed0d80365c2d5"
server_key_iv: "a78bfc64b156352f7804cff7c2a69f5e"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:18.343657 31041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 14:53:18.370563 31057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:18.372244 31041 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260619 14:53:18.372483 31041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "031be3ff9dd144b49fc3c01641a1cd48"
format_stamp: "Formatted at 2026-06-19 14:53:18 on dist-test-slave-3qdh"
server_key: "6213d016a670be737c6ed0d80365c2d5"
server_key_iv: "a78bfc64b156352f7804cff7c2a69f5e"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:18.372692 31041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:53:18.396880 31041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:53:18.400189 31041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:53:18.400439 31041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:53:18.401149 31041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:53:18.402185 31041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:53:18.402316 31041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:18.402411 31041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:53:18.402473 31041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:18.414259 31041 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:38783
I20260619 14:53:18.414294 31170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:38783 every 8 connection(s)
I20260619 14:53:18.415187 31041 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:53:18.419953 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 31041
I20260619 14:53:18.420109 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:53:18.420482 19593 external_mini_cluster.cc:1428] Setting key 4839fa3c8c5a94595644faf2294fe8ff
Jun 19 14:53:18 dist-test-slave-3qdh krb5kdc[30345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880798, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:53:18.434005 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:18.417021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:44135 (local address 127.19.34.126:45541)
0619 14:53:18.417185 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:18.417190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:18.418031 (+ 841us) server_negotiation.cc:407] Connection header received
0619 14:53:18.419096 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:18.419099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:18.419146 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:18.419226 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:18.421142 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.421710 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.423252 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.423678 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.423777 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:18.426390 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:18.426394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:18.426396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:18.426423 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:18.428087 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:18.432106 (+ 4019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:18.432111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:18.432113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:18.432215 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:18.432701 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:18.432705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:18.432707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:18.432935 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:18.433029 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:18.433698 (+ 669us) server_negotiation.cc:299] Negotiation successful
0619 14:53:18.433835 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 14:53:18.435657 31173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:18.417404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45541 (local address 127.19.34.66:44135)
0619 14:53:18.417864 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:18.417915 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:18.418863 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:18.419849 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:18.419859 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:18.420225 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:53:18.420989 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:18.421011 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.421851 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.421855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:18.423097 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:18.423112 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.423988 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.423995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:18.424222 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:18.424903 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:53:18.424920 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:53:18.426225 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:53:18.431566 (+ 5341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:18.431572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:18.431586 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:18.431965 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:18.432380 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:18.432383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:18.432385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:18.432573 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:18.433104 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:53:18.433110 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:53:18.433396 (+ 286us) client_negotiation.cc:769] Sending connection context
0619 14:53:18.434419 (+ 1023us) client_negotiation.cc:240] Negotiation successful
0619 14:53:18.434740 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":116,"threads_started":1}
I20260619 14:53:18.437024 31171 heartbeater.cc:347] Connected to a master server at 127.19.34.126:45541
I20260619 14:53:18.437232 31171 heartbeater.cc:464] Registering TS with master...
I20260619 14:53:18.437752 31171 heartbeater.cc:511] Master 127.19.34.126:45541 requested a full tablet report, sending...
I20260619 14:53:18.438877 30821 ts_manager.cc:194] Registered new tserver with Master: 031be3ff9dd144b49fc3c01641a1cd48 (127.19.34.66:38783)
I20260619 14:53:18.439633 30821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:44135
WARNING: no policy specified for kudu/127.19.34.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
I20260619 14:53:18.461386 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.34.84:59307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:45541
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:45521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:53:18 dist-test-slave-3qdh krb5kdc[30345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880798, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
W20260619 14:53:18.599975 31176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:53:18.600260 31176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:53:18.600334 31176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:53:18.605487 31176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:53:18.605609 31176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:53:18.605717 31176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:53:18.612485 31176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:45521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.34.84:59307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:45541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:53:18.613445 31176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:53:18.614459 31176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:53:18.624599 31182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:18.624877 31184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:18.624912 31176 server_base.cc:1034] running on GCE node
W20260619 14:53:18.625113 31181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:18.625469 31176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:53:18.626037 31176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:53:18.627223 31176 hybrid_clock.cc:648] HybridClock initialized: now 1781880798627201 us; error 35 us; skew 500 ppm
Jun 19 14:53:18 dist-test-slave-3qdh krb5kdc[30345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880798, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:53:18.630290 31176 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:53:18.631224 31176 webserver.cc:460] Webserver started at http://127.19.34.67:41613/ using document root <none> and password file <none>
I20260619 14:53:18.631879 31176 fs_manager.cc:359] Metadata directory not provided
I20260619 14:53:18.631973 31176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:53:18.632231 31176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:53:18 dist-test-slave-3qdh krb5kdc[30345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880798, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
I20260619 14:53:18.646299 31176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "aee841e8672f4cf5b0504cbb612be2ae"
format_stamp: "Formatted at 2026-06-19 14:53:18 on dist-test-slave-3qdh"
server_key: "c75d5e767729dff7fe994d491f772150"
server_key_iv: "21319b573be09d2d6a9fb15bec8f138c"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:18.647222 31176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "aee841e8672f4cf5b0504cbb612be2ae"
format_stamp: "Formatted at 2026-06-19 14:53:18 on dist-test-slave-3qdh"
server_key: "c75d5e767729dff7fe994d491f772150"
server_key_iv: "21319b573be09d2d6a9fb15bec8f138c"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:18.657569 31176 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.003s sys 0.000s
I20260619 14:53:18.672206 31191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:18.674197 31176 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:53:18.674320 31176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "aee841e8672f4cf5b0504cbb612be2ae"
format_stamp: "Formatted at 2026-06-19 14:53:18 on dist-test-slave-3qdh"
server_key: "c75d5e767729dff7fe994d491f772150"
server_key_iv: "21319b573be09d2d6a9fb15bec8f138c"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:18.674443 31176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:53:18.688360 31176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:53:18.691634 31176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:53:18.691839 31176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:53:18.692431 31176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:53:18.693346 31176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:53:18.693408 31176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:18.693507 31176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:53:18.693559 31176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:18.701588 31176 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:36393
I20260619 14:53:18.702742 31176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:53:18.703773 31304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:36393 every 8 connection(s)
I20260619 14:53:18.710083 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 31176
I20260619 14:53:18.710187 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:53:18.710464 19593 external_mini_cluster.cc:1428] Setting key ed77745c5d03f5ddd4b36763355d0b7a
Jun 19 14:53:18 dist-test-slave-3qdh krb5kdc[30345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880798, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:53:18.715778 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:18.705064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:53139 (local address 127.19.34.126:45541)
0619 14:53:18.705168 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:18.705171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:18.705924 (+ 753us) server_negotiation.cc:407] Connection header received
0619 14:53:18.706953 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:18.706956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:18.707010 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:18.707141 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:18.708118 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.708685 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.709652 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.709862 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.709952 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:18.712671 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:18.712676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:18.712678 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:18.712701 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:18.713961 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:18.714476 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:18.714481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:18.714483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:18.714566 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:18.714856 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:18.714860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:18.714861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:18.715006 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:18.715100 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:18.715547 (+ 447us) server_negotiation.cc:299] Negotiation successful
0619 14:53:18.715640 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:53:18.716575 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:18.705403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45541 (local address 127.19.34.67:53139)
0619 14:53:18.705796 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:18.705840 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:18.706759 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:18.707320 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:18.707330 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:18.707556 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:53:18.707971 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:18.707984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.708801 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.708808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:18.709539 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:18.709548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.710100 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.710105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:18.710273 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:18.710782 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:53:18.710801 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:53:18.712508 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:53:18.714078 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:18.714083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:18.714093 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:18.714381 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:18.714647 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:18.714651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:18.714652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:18.714772 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:18.715108 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:53:18.715115 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:53:18.715304 (+ 189us) client_negotiation.cc:769] Sending connection context
0619 14:53:18.715506 (+ 202us) client_negotiation.cc:240] Negotiation successful
0619 14:53:18.715739 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":129,"threads_started":1}
I20260619 14:53:18.717456 31305 heartbeater.cc:347] Connected to a master server at 127.19.34.126:45541
I20260619 14:53:18.717638 31305 heartbeater.cc:464] Registering TS with master...
I20260619 14:53:18.718016 31305 heartbeater.cc:511] Master 127.19.34.126:45541 requested a full tablet report, sending...
I20260619 14:53:18.718840 30821 ts_manager.cc:194] Registered new tserver with Master: aee841e8672f4cf5b0504cbb612be2ae (127.19.34.67:36393)
I20260619 14:53:18.719307 30821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:53139
I20260619 14:53:18.724095 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:53:18.732689 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:18.725820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49698 (local address 127.19.34.126:45541)
0619 14:53:18.725926 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:18.725930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:18.726184 (+ 254us) server_negotiation.cc:407] Connection header received
0619 14:53:18.726297 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:18.726300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:18.726337 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:18.726426 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:18.727094 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.727530 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.728398 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.728609 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.728709 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:18.729837 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:18.729840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:18.729842 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:18.729862 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:18.731135 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:18.731551 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:18.731554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:18.731556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:18.731633 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:18.731899 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:18.731903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:18.731905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:18.732062 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:18.732203 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:18.732447 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 14:53:18.732537 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:53:18.735380 30821 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:53:18.738608 30821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:53:18.754416 31318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:18.747011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.19.34.66:38783)
0619 14:53:18.747768 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:18.747798 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:18.748711 (+ 913us) server_negotiation.cc:407] Connection header received
0619 14:53:18.748913 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:18.748919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:18.749121 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:18.749270 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:53:18.749700 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.750866 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.752835 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.753611 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.753678 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:18.753803 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:18.754177 (+ 374us) server_negotiation.cc:299] Negotiation successful
0619 14:53:18.754272 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":101,"threads_started":1}
I20260619 14:53:18.754974 31319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:18.746290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:38783 (local address 127.0.0.1:55350)
0619 14:53:18.748569 (+ 2279us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:18.748611 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:18.748758 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:18.749422 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:18.749428 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:18.749447 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:53:18.749580 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:18.749586 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.751309 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.751312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:18.752690 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:18.752700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.753888 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.753891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:18.753993 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:18.754024 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 14:53:18.754763 (+ 739us) client_negotiation.cc:240] Negotiation successful
0619 14:53:18.754861 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2125,"mutex_wait_us":3,"thread_start_us":86,"threads_started":1}
I20260619 14:53:18.757783 31090 tablet_service.cc:1467] Processing CreateTablet for tablet ecbe941d2a144dfa916bd8ec8639a473 (DEFAULT_TABLE table=test-table [id=61f7bd405ac64b5f8e706ace704c491e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:53:18.758800 31090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecbe941d2a144dfa916bd8ec8639a473. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:53:18.759464 31321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:18.747102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:36393 (local address 127.0.0.1:42474)
0619 14:53:18.748984 (+ 1882us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:18.748995 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:18.749133 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:18.749691 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:18.749693 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:18.749708 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:53:18.749790 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:18.749795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.751088 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.751091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:18.757351 (+ 6260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:18.757359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.759116 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.759119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:18.759225 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:18.759243 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:53:18.759304 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 14:53:18.759357 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1830,"mutex_wait_us":4663,"thread_start_us":59,"threads_started":1}
I20260619 14:53:18.759845 31320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:18.747399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42474 (local address 127.19.34.67:36393)
0619 14:53:18.747956 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:18.747966 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:18.749072 (+ 1106us) server_negotiation.cc:407] Connection header received
0619 14:53:18.749269 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:18.749273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:18.749440 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:18.749543 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:53:18.749896 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.750939 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.757473 (+ 6534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.758803 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.758876 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:18.758991 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:18.759648 (+ 657us) server_negotiation.cc:299] Negotiation successful
0619 14:53:18.759727 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":150,"threads_started":1}
I20260619 14:53:18.762367 31239 tablet_service.cc:1467] Processing CreateTablet for tablet ecbe941d2a144dfa916bd8ec8639a473 (DEFAULT_TABLE table=test-table [id=61f7bd405ac64b5f8e706ace704c491e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:53:18.763234 31239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecbe941d2a144dfa916bd8ec8639a473. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:53:18.770970 31324 tablet_bootstrap.cc:492] T ecbe941d2a144dfa916bd8ec8639a473 P aee841e8672f4cf5b0504cbb612be2ae: Bootstrap starting.
I20260619 14:53:18.772034 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:18.747480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:41139 (local address 127.0.0.1:42254)
0619 14:53:18.752748 (+ 5268us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:18.752761 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:18.752855 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:18.753445 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:18.753447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:18.753462 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:53:18.753565 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:18.753571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.755391 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.755393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:18.756330 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:18.756339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.771605 (+ 15266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.771608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:18.771710 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:18.771741 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 14:53:18.771817 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:53:18.771890 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5205,"thread_start_us":79,"threads_started":1}
I20260619 14:53:18.772601 31322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:18.747925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42254 (local address 127.19.34.65:41139)
0619 14:53:18.748142 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:18.748165 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:18.752938 (+ 4773us) server_negotiation.cc:407] Connection header received
0619 14:53:18.753024 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:18.753028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:18.753198 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:18.753328 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:53:18.753690 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.755286 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.756446 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.757081 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.757167 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:18.757277 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:18.772373 (+ 15096us) server_negotiation.cc:299] Negotiation successful
0619 14:53:18.772487 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260619 14:53:18.772578 31323 tablet_bootstrap.cc:492] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48: Bootstrap starting.
I20260619 14:53:18.773689 31324 tablet_bootstrap.cc:654] T ecbe941d2a144dfa916bd8ec8639a473 P aee841e8672f4cf5b0504cbb612be2ae: Neither blocks nor log segments found. Creating new log.
I20260619 14:53:18.774459 31324 log.cc:824] T ecbe941d2a144dfa916bd8ec8639a473 P aee841e8672f4cf5b0504cbb612be2ae: Log is configured to *not* fsync() on all Append() calls
I20260619 14:53:18.774883 31323 tablet_bootstrap.cc:654] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48: Neither blocks nor log segments found. Creating new log.
I20260619 14:53:18.775254 30969 tablet_service.cc:1467] Processing CreateTablet for tablet ecbe941d2a144dfa916bd8ec8639a473 (DEFAULT_TABLE table=test-table [id=61f7bd405ac64b5f8e706ace704c491e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:53:18.775575 31323 log.cc:824] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48: Log is configured to *not* fsync() on all Append() calls
I20260619 14:53:18.776077 30969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecbe941d2a144dfa916bd8ec8639a473. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:53:18.781427 31327 tablet_bootstrap.cc:492] T ecbe941d2a144dfa916bd8ec8639a473 P edaefb4f27c94f3585b9314461f8f855: Bootstrap starting.
I20260619 14:53:18.783078 31327 tablet_bootstrap.cc:654] T ecbe941d2a144dfa916bd8ec8639a473 P edaefb4f27c94f3585b9314461f8f855: Neither blocks nor log segments found. Creating new log.
I20260619 14:53:18.783627 31327 log.cc:824] T ecbe941d2a144dfa916bd8ec8639a473 P edaefb4f27c94f3585b9314461f8f855: Log is configured to *not* fsync() on all Append() calls
I20260619 14:53:18.785413 31323 tablet_bootstrap.cc:492] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48: No bootstrap required, opened a new log
I20260619 14:53:18.785627 31323 ts_tablet_manager.cc:1397] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260619 14:53:18.785848 31327 tablet_bootstrap.cc:492] T ecbe941d2a144dfa916bd8ec8639a473 P edaefb4f27c94f3585b9314461f8f855: No bootstrap required, opened a new log
I20260619 14:53:18.785902 31324 tablet_bootstrap.cc:492] T ecbe941d2a144dfa916bd8ec8639a473 P aee841e8672f4cf5b0504cbb612be2ae: No bootstrap required, opened a new log
I20260619 14:53:18.786038 31327 ts_tablet_manager.cc:1397] T ecbe941d2a144dfa916bd8ec8639a473 P edaefb4f27c94f3585b9314461f8f855: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:53:18.786047 31324 ts_tablet_manager.cc:1397] T ecbe941d2a144dfa916bd8ec8639a473 P aee841e8672f4cf5b0504cbb612be2ae: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260619 14:53:18.789108 31323 raft_consensus.cc:348] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031be3ff9dd144b49fc3c01641a1cd48" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 38783 } } peers { permanent_uuid: "edaefb4f27c94f3585b9314461f8f855" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41139 } } peers { permanent_uuid: "aee841e8672f4cf5b0504cbb612be2ae" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 36393 } }
I20260619 14:53:18.789328 31323 raft_consensus.cc:374] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:53:18.789368 31323 raft_consensus.cc:729] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031be3ff9dd144b49fc3c01641a1cd48, State: Initialized, Role: FOLLOWER
I20260619 14:53:18.789322 31324 raft_consensus.cc:348] T ecbe941d2a144dfa916bd8ec8639a473 P aee841e8672f4cf5b0504cbb612be2ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031be3ff9dd144b49fc3c01641a1cd48" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 38783 } } peers { permanent_uuid: "edaefb4f27c94f3585b9314461f8f855" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41139 } } peers { permanent_uuid: "aee841e8672f4cf5b0504cbb612be2ae" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 36393 } }
I20260619 14:53:18.789531 31324 raft_consensus.cc:374] T ecbe941d2a144dfa916bd8ec8639a473 P aee841e8672f4cf5b0504cbb612be2ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:53:18.789438 31327 raft_consensus.cc:348] T ecbe941d2a144dfa916bd8ec8639a473 P edaefb4f27c94f3585b9314461f8f855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031be3ff9dd144b49fc3c01641a1cd48" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 38783 } } peers { permanent_uuid: "edaefb4f27c94f3585b9314461f8f855" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41139 } } peers { permanent_uuid: "aee841e8672f4cf5b0504cbb612be2ae" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 36393 } }
I20260619 14:53:18.789572 31324 raft_consensus.cc:729] T ecbe941d2a144dfa916bd8ec8639a473 P aee841e8672f4cf5b0504cbb612be2ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee841e8672f4cf5b0504cbb612be2ae, State: Initialized, Role: FOLLOWER
I20260619 14:53:18.789614 31327 raft_consensus.cc:374] T ecbe941d2a144dfa916bd8ec8639a473 P edaefb4f27c94f3585b9314461f8f855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:53:18.789656 31327 raft_consensus.cc:729] T ecbe941d2a144dfa916bd8ec8639a473 P edaefb4f27c94f3585b9314461f8f855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edaefb4f27c94f3585b9314461f8f855, State: Initialized, Role: FOLLOWER
I20260619 14:53:18.789772 31323 consensus_queue.cc:260] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031be3ff9dd144b49fc3c01641a1cd48" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 38783 } } peers { permanent_uuid: "edaefb4f27c94f3585b9314461f8f855" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41139 } } peers { permanent_uuid: "aee841e8672f4cf5b0504cbb612be2ae" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 36393 } }
I20260619 14:53:18.789969 31327 consensus_queue.cc:260] T ecbe941d2a144dfa916bd8ec8639a473 P edaefb4f27c94f3585b9314461f8f855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031be3ff9dd144b49fc3c01641a1cd48" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 38783 } } peers { permanent_uuid: "edaefb4f27c94f3585b9314461f8f855" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41139 } } peers { permanent_uuid: "aee841e8672f4cf5b0504cbb612be2ae" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 36393 } }
I20260619 14:53:18.790588 31324 consensus_queue.cc:260] T ecbe941d2a144dfa916bd8ec8639a473 P aee841e8672f4cf5b0504cbb612be2ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031be3ff9dd144b49fc3c01641a1cd48" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 38783 } } peers { permanent_uuid: "edaefb4f27c94f3585b9314461f8f855" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41139 } } peers { permanent_uuid: "aee841e8672f4cf5b0504cbb612be2ae" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 36393 } }
I20260619 14:53:18.790675 31323 ts_tablet_manager.cc:1428] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:53:18.790980 31327 ts_tablet_manager.cc:1428] T ecbe941d2a144dfa916bd8ec8639a473 P edaefb4f27c94f3585b9314461f8f855: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:53:18.791335 31035 heartbeater.cc:503] Master 127.19.34.126:45541 was elected leader, sending a full tablet report...
I20260619 14:53:18.791492 31324 ts_tablet_manager.cc:1428] T ecbe941d2a144dfa916bd8ec8639a473 P aee841e8672f4cf5b0504cbb612be2ae: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:53:18.791754 31305 heartbeater.cc:503] Master 127.19.34.126:45541 was elected leader, sending a full tablet report...
I20260619 14:53:18.795907 31171 heartbeater.cc:503] Master 127.19.34.126:45541 was elected leader, sending a full tablet report...
I20260619 14:53:18.804886 31329 raft_consensus.cc:482] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:53:18.805061 31329 raft_consensus.cc:504] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031be3ff9dd144b49fc3c01641a1cd48" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 38783 } } peers { permanent_uuid: "edaefb4f27c94f3585b9314461f8f855" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41139 } } peers { permanent_uuid: "aee841e8672f4cf5b0504cbb612be2ae" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 36393 } }
I20260619 14:53:18.810393 31173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:18.806141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:41139 (local address 127.19.34.66:52581)
0619 14:53:18.806340 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:18.806353 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:18.806424 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:18.806907 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:18.806910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:18.806928 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:53:18.807027 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:18.807033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.808004 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.808008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:18.809158 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:18.809170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.809997 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.810000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:18.810103 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:18.810124 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:53:18.810188 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 14:53:18.810253 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260619 14:53:18.810963 31322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:18.806547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:52581 (local address 127.19.34.65:41139)
0619 14:53:18.806639 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:18.806644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:18.806654 (+ 10us) server_negotiation.cc:407] Connection header received
0619 14:53:18.806689 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:18.806691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:18.806744 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:18.806807 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:53:18.807158 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.807880 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.809290 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.809801 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.809903 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:18.809945 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:18.810752 (+ 807us) server_negotiation.cc:299] Negotiation successful
0619 14:53:18.810828 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:53:18.811623 30984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecbe941d2a144dfa916bd8ec8639a473" candidate_uuid: "031be3ff9dd144b49fc3c01641a1cd48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edaefb4f27c94f3585b9314461f8f855" is_pre_election: true
I20260619 14:53:18.811870 30984 raft_consensus.cc:2454] T ecbe941d2a144dfa916bd8ec8639a473 P edaefb4f27c94f3585b9314461f8f855 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031be3ff9dd144b49fc3c01641a1cd48 in term 0.
I20260619 14:53:18.812700 31061 leader_election.cc:302] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031be3ff9dd144b49fc3c01641a1cd48, edaefb4f27c94f3585b9314461f8f855; no voters:
I20260619 14:53:18.817395 31173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:18.813377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:36393 (local address 127.19.34.66:35609)
0619 14:53:18.813805 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:18.813820 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:18.813894 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:18.814301 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:18.814303 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:18.814318 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:53:18.814381 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:18.814386 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.815300 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.815305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:18.816210 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:18.816220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.817078 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.817081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:18.817158 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:18.817185 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:53:18.817251 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 14:53:18.817303 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361}
I20260619 14:53:18.817699 31320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:18.813610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:35609 (local address 127.19.34.67:36393)
0619 14:53:18.813720 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:18.813726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:18.813980 (+ 254us) server_negotiation.cc:407] Connection header received
0619 14:53:18.814093 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:18.814096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:18.814134 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:18.814197 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:53:18.814497 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.815162 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.816369 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.816863 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.816966 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:18.817005 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:18.817557 (+ 552us) server_negotiation.cc:299] Negotiation successful
0619 14:53:18.817621 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:53:18.818265 31259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecbe941d2a144dfa916bd8ec8639a473" candidate_uuid: "031be3ff9dd144b49fc3c01641a1cd48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee841e8672f4cf5b0504cbb612be2ae" is_pre_election: true
I20260619 14:53:18.818472 31259 raft_consensus.cc:2454] T ecbe941d2a144dfa916bd8ec8639a473 P aee841e8672f4cf5b0504cbb612be2ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031be3ff9dd144b49fc3c01641a1cd48 in term 0.
I20260619 14:53:18.818020 31329 leader_election.cc:288] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edaefb4f27c94f3585b9314461f8f855 (127.19.34.65:41139), aee841e8672f4cf5b0504cbb612be2ae (127.19.34.67:36393)
I20260619 14:53:18.818742 31329 raft_consensus.cc:2781] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:53:18.818866 31329 raft_consensus.cc:482] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:53:18.818948 31329 raft_consensus.cc:3037] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:53:18.820093 31329 raft_consensus.cc:504] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031be3ff9dd144b49fc3c01641a1cd48" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 38783 } } peers { permanent_uuid: "edaefb4f27c94f3585b9314461f8f855" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41139 } } peers { permanent_uuid: "aee841e8672f4cf5b0504cbb612be2ae" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 36393 } }
I20260619 14:53:18.820598 30984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecbe941d2a144dfa916bd8ec8639a473" candidate_uuid: "031be3ff9dd144b49fc3c01641a1cd48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edaefb4f27c94f3585b9314461f8f855"
I20260619 14:53:18.820688 30984 raft_consensus.cc:3037] T ecbe941d2a144dfa916bd8ec8639a473 P edaefb4f27c94f3585b9314461f8f855 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:53:18.821842 30984 raft_consensus.cc:2454] T ecbe941d2a144dfa916bd8ec8639a473 P edaefb4f27c94f3585b9314461f8f855 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031be3ff9dd144b49fc3c01641a1cd48 in term 1.
I20260619 14:53:18.822149 31061 leader_election.cc:302] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031be3ff9dd144b49fc3c01641a1cd48, edaefb4f27c94f3585b9314461f8f855; no voters:
I20260619 14:53:18.822352 31329 leader_election.cc:288] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 [CANDIDATE]: Term 1 election: Requested vote from peers edaefb4f27c94f3585b9314461f8f855 (127.19.34.65:41139), aee841e8672f4cf5b0504cbb612be2ae (127.19.34.67:36393)
I20260619 14:53:18.822480 31329 raft_consensus.cc:2781] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:53:18.822590 31329 raft_consensus.cc:686] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 [term 1 LEADER]: Becoming Leader. State: Replica: 031be3ff9dd144b49fc3c01641a1cd48, State: Running, Role: LEADER
I20260619 14:53:18.822664 31259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecbe941d2a144dfa916bd8ec8639a473" candidate_uuid: "031be3ff9dd144b49fc3c01641a1cd48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee841e8672f4cf5b0504cbb612be2ae"
I20260619 14:53:18.822746 31259 raft_consensus.cc:3037] T ecbe941d2a144dfa916bd8ec8639a473 P aee841e8672f4cf5b0504cbb612be2ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:53:18.822889 31329 consensus_queue.cc:237] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031be3ff9dd144b49fc3c01641a1cd48" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 38783 } } peers { permanent_uuid: "edaefb4f27c94f3585b9314461f8f855" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41139 } } peers { permanent_uuid: "aee841e8672f4cf5b0504cbb612be2ae" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 36393 } }
I20260619 14:53:18.823773 31259 raft_consensus.cc:2454] T ecbe941d2a144dfa916bd8ec8639a473 P aee841e8672f4cf5b0504cbb612be2ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031be3ff9dd144b49fc3c01641a1cd48 in term 1.
I20260619 14:53:18.825412 30821 catalog_manager.cc:5495] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031be3ff9dd144b49fc3c01641a1cd48 (127.19.34.66). New cstate: current_term: 1 leader_uuid: "031be3ff9dd144b49fc3c01641a1cd48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031be3ff9dd144b49fc3c01641a1cd48" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 38783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edaefb4f27c94f3585b9314461f8f855" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aee841e8672f4cf5b0504cbb612be2ae" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 36393 } health_report { overall_health: UNKNOWN } } }
Jun 19 14:53:18 dist-test-slave-3qdh krb5kdc[30345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880798, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
I20260619 14:53:18.844586 31318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:18.839923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55354 (local address 127.19.34.66:38783)
0619 14:53:18.840283 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:18.840287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:18.840298 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:53:18.840383 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:18.840386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:18.840442 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:18.840532 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:53:18.840925 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.841723 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.843079 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:18.843373 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:18.843458 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:18.843831 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:53:18.844197 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:53:18.844370 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:18.844401 (+ 31us) server_negotiation.cc:299] Negotiation successful
0619 14:53:18.844444 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291}
I20260619 14:53:18.850570 31259 raft_consensus.cc:1261] T ecbe941d2a144dfa916bd8ec8639a473 P aee841e8672f4cf5b0504cbb612be2ae [term 1 FOLLOWER]: Refusing update from remote peer 031be3ff9dd144b49fc3c01641a1cd48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:53:18.851678 30984 raft_consensus.cc:1261] T ecbe941d2a144dfa916bd8ec8639a473 P edaefb4f27c94f3585b9314461f8f855 [term 1 FOLLOWER]: Refusing update from remote peer 031be3ff9dd144b49fc3c01641a1cd48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:53:18.852336 31329 consensus_queue.cc:1035] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aee841e8672f4cf5b0504cbb612be2ae" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 36393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:53:18.852679 31329 consensus_queue.cc:1035] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edaefb4f27c94f3585b9314461f8f855" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:53:18.868513 31336 mvcc.cc:205] Tried to move back new op lower bound from 7298583752083755008 to 7298583751980986368. Current Snapshot: MvccSnapshot[applied={T|T < 7298583752083755008}]
I20260619 14:53:18.869742 31339 mvcc.cc:205] Tried to move back new op lower bound from 7298583752083755008 to 7298583751980986368. Current Snapshot: MvccSnapshot[applied={T|T < 7298583752083755008}]
I20260619 14:53:18.873044 31337 mvcc.cc:205] Tried to move back new op lower bound from 7298583752083755008 to 7298583751980986368. Current Snapshot: MvccSnapshot[applied={T|T < 7298583752083755008}]
I20260619 14:53:18.876960 30821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:53:18.877106 30821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:53:18.880053 30821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49689aef3a874060b1d1a817f357b7ce: Sending DeleteTablet for 3 replicas of tablet ecbe941d2a144dfa916bd8ec8639a473
I20260619 14:53:18.881819 31239 tablet_service.cc:1514] Processing DeleteTablet for tablet ecbe941d2a144dfa916bd8ec8639a473 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:53:18 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:42474
I20260619 14:53:18.882241 31090 tablet_service.cc:1514] Processing DeleteTablet for tablet ecbe941d2a144dfa916bd8ec8639a473 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:53:18 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:55350
I20260619 14:53:18.881588 30969 tablet_service.cc:1514] Processing DeleteTablet for tablet ecbe941d2a144dfa916bd8ec8639a473 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:53:18 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:42254
I20260619 14:53:18.883569 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 30902
I20260619 14:53:18.887966 31346 tablet_replica.cc:330] T ecbe941d2a144dfa916bd8ec8639a473 P aee841e8672f4cf5b0504cbb612be2ae: stopping tablet replica
I20260619 14:53:18.888163 31347 tablet_replica.cc:330] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48: stopping tablet replica
I20260619 14:53:18.890295 31347 raft_consensus.cc:2229] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:53:18.890586 31346 raft_consensus.cc:2229] T ecbe941d2a144dfa916bd8ec8639a473 P aee841e8672f4cf5b0504cbb612be2ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:53:18.890868 31346 raft_consensus.cc:2258] T ecbe941d2a144dfa916bd8ec8639a473 P aee841e8672f4cf5b0504cbb612be2ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:53:18.891229 31347 raft_consensus.cc:2258] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:53:18.892881 31347 ts_tablet_manager.cc:1905] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:53:18.893138 31346 ts_tablet_manager.cc:1905] T ecbe941d2a144dfa916bd8ec8639a473 P aee841e8672f4cf5b0504cbb612be2ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:53:18.895365 31347 ts_tablet_manager.cc:1918] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:53:18.895512 31347 log.cc:1196] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-1/wal/wals/ecbe941d2a144dfa916bd8ec8639a473
I20260619 14:53:18.895874 31347 ts_tablet_manager.cc:1939] T ecbe941d2a144dfa916bd8ec8639a473 P 031be3ff9dd144b49fc3c01641a1cd48: Deleting consensus metadata
I20260619 14:53:18.895972 31346 ts_tablet_manager.cc:1918] T ecbe941d2a144dfa916bd8ec8639a473 P aee841e8672f4cf5b0504cbb612be2ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:53:18.897020 30807 catalog_manager.cc:4841] TS 031be3ff9dd144b49fc3c01641a1cd48 (127.19.34.66:38783): tablet ecbe941d2a144dfa916bd8ec8639a473 (table test-table [id=61f7bd405ac64b5f8e706ace704c491e]) successfully deleted
W20260619 14:53:18.898337 30808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.34.65:41139 (error 108)
I20260619 14:53:18.900488 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 31041
I20260619 14:53:18.901839 31346 log.cc:1196] T ecbe941d2a144dfa916bd8ec8639a473 P aee841e8672f4cf5b0504cbb612be2ae: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/ecbe941d2a144dfa916bd8ec8639a473
I20260619 14:53:18.902237 31346 ts_tablet_manager.cc:1939] T ecbe941d2a144dfa916bd8ec8639a473 P aee841e8672f4cf5b0504cbb612be2ae: Deleting consensus metadata
I20260619 14:53:18.903110 30806 catalog_manager.cc:4841] TS aee841e8672f4cf5b0504cbb612be2ae (127.19.34.67:36393): tablet ecbe941d2a144dfa916bd8ec8639a473 (table test-table [id=61f7bd405ac64b5f8e706ace704c491e]) successfully deleted
I20260619 14:53:18.905462 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:18.904647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:41139 (local address 127.0.0.1:42264)
0619 14:53:18.905347 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:18.905399 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.65:41139: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":616}
W20260619 14:53:18.905642 30808 catalog_manager.cc:4584] TS edaefb4f27c94f3585b9314461f8f855 (127.19.34.65:41139): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ecbe941d2a144dfa916bd8ec8639a473: Network error: Client connection negotiation failed: client connection to 127.19.34.65:41139: connect: Connection refused (error 111)
I20260619 14:53:18.907641 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 31176
I20260619 14:53:18.914038 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 30789
I20260619 14:53:18.922992 19593 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 14:53:19.258558 19593 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 14:53:19.258673 19593 mini_ranger.cc:66] Stopping Ranger...
I20260619 14:53:19.602669 19593 mini_ranger.cc:68] Stopped Ranger
2026-06-19 14:53:19.602 UTC [30375] LOG: received smart shutdown request
2026-06-19 14:53:19.605 UTC [30375] LOG: background worker "logical replication launcher" (PID 30384) exited with exit code 1
2026-06-19 14:53:19.606 UTC [30379] LOG: shutting down
2026-06-19 14:53:19.737 UTC [30375] LOG: database system is shut down
2026-06-19T14:53:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (42361 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:53:19 dist-test-slave-3qdh krb5kdc[31359](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:53:19 dist-test-slave-3qdh krb5kdc[31359](info): set up 2 sockets
Jun 19 14:53:19 dist-test-slave-3qdh krb5kdc[31359](info): commencing operation
krb5kdc: starting...
W20260619 14:53:20.804190 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:53:20 dist-test-slave-3qdh krb5kdc[31359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:53:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:53:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.34.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.34.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.34.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/rangeradmin_127.19.34.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.34.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/rangeradmin_127.19.34.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.34.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.34.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.34.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/rangerlookup_127.19.34.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.34.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/rangerlookup_127.19.34.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.34.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.34.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.34.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/HTTP_127.19.34.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.34.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/HTTP_127.19.34.84@KRBTEST.COM.keytab.
I20260619 14:53:20.944980 19593 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0Qcekt/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/postgres -l logfile start
2026-06-19 14:53:23.103 UTC [31389] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 14:53:23.103 UTC [31389] LOG: listening on IPv4 address "127.19.34.84", port 39105
2026-06-19 14:53:23.106 UTC [31389] LOG: listening on Unix socket "/tmp/.s.PGSQL.39105"
2026-06-19 14:53:23.140 UTC [31392] LOG: database system was shut down at 2026-06-19 14:53:21 UTC
2026-06-19 14:53:23.144 UTC [31389] LOG: database system is ready to accept connections
I20260619 14:53:24.047919 19593 mini_postgres.cc:96] Postgres bound to 39105
2026-06-19 14:53:24.053 UTC [31400] FATAL: database "slave" does not exist
127.19.34.84:39105 - accepting connections
I20260619 14:53:24.054262 19593 mini_ranger.cc:161] Starting Ranger...
I20260619 14:53:24.067932 19593 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 14:53:24.454237 19593 mini_ranger.cc:87] Created ranger Postgres database
I20260619 14:53:24.454313 19593 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ranger-admin
2026-06-19 14:53:24,696 [I] DB FLAVOR :POSTGRES
2026-06-19 14:53:24,697 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:53:24,697 [I] Checking connection..
2026-06-19 14:53:24,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 14:53:24,912 [I] Checking connection passed.
2026-06-19 14:53:24,912 [I] --------- Verifying version history table ---------
2026-06-19 14:53:24,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:53:25,129 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:53:25,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:53:25,333 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:53:25,334 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 14:53:25,334 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 14:53:25.533 UTC [31464] WARNING: there is no transaction in progress
2026-06-19 14:53:25,539 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 14:53:25,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:53:25,751 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 14:53:25,751 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:53:25,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 14:53:25,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 14:53:26,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-3qdh.c.gcp-upstream.internal','N') ;"
2026-06-19 14:53:26,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:53:26,568 [I] Table x_portal_user does not exist in database ranger
2026-06-19 14:53:26,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:53:26,791 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 14:53:26,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:53:26,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:53:27,205 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 14:53:27,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 14:53:28.000 UTC [31572] WARNING: there is no transaction in progress
2026-06-19 14:53:28.014 UTC [31572] WARNING: there is no transaction in progress
2026-06-19 14:53:28.028 UTC [31572] WARNING: there is no transaction in progress
2026-06-19 14:53:28.041 UTC [31572] WARNING: there is no transaction in progress
2026-06-19 14:53:28.058 UTC [31572] WARNING: there is no transaction in progress
2026-06-19 14:53:28.071 UTC [31572] WARNING: there is no transaction in progress
2026-06-19 14:53:28.117 UTC [31572] WARNING: there is no transaction in progress
2026-06-19 14:53:28.122 UTC [31572] WARNING: there is no transaction in progress
2026-06-19 14:53:28.135 UTC [31572] WARNING: there is no transaction in progress
2026-06-19 14:53:28.143 UTC [31572] WARNING: there is no transaction in progress
2026-06-19 14:53:28.152 UTC [31572] WARNING: there is no transaction in progress
2026-06-19 14:53:28.164 UTC [31572] WARNING: there is no transaction in progress
2026-06-19 14:53:28.172 UTC [31572] WARNING: there is no transaction in progress
2026-06-19 14:53:28.178 UTC [31572] WARNING: there is no transaction in progress
2026-06-19 14:53:28.183 UTC [31572] WARNING: there is no transaction in progress
2026-06-19 14:53:28.188 UTC [31572] WARNING: there is no transaction in progress
2026-06-19 14:53:28,709 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 14:53:28,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 14:53:28,931 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 14:53:28,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:53:29,161 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 14:53:29,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:53:29,368 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 14:53:29,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:53:29,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:53:29,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-3qdh.c.gcp-upstream.internal';"
2026-06-19 14:53:30,042 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 14:53:30,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 14:53:30,234 [I] DB_PATCHES have already been applied
I20260619 14:53:30.239373 19593 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ranger-admin:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0Qcekt/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 14:53:30.239424 19593 mini_ranger.cc:193] Using host: 127.19.34.84
I20260619 14:53:30.242816 19593 mini_ranger.cc:227] Ranger admin URL: http://127.19.34.84:41423
Jun 19, 2026 2:53:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:53:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:53:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:53:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:53:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:53:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:53:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 14:53:31.241004 19593 mini_ranger.cc:160] Time spent starting Ranger: real 7.187s user 0.000s sys 0.009s
Jun 19, 2026 2:53:48 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:53:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 19, 2026 2:53:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:53:48 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:53:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:53:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:53:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:53:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:53:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:53:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:53:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:53:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:53:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:53:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:53:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:53:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:53:48 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:53:49 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 14:53:50.081467 19593 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.34.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.34.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.34.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/rangerkms_127.19.34.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.34.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/rangerkms_127.19.34.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.34.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.34.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.34.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/HTTP_127.19.34.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.34.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/HTTP_127.19.34.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 14:53:50 dist-test-slave-3qdh krb5kdc[31359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880830, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 14:53:50.155711 19593 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 14:53:50.164060 19593 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 14:53:50.910923 19593 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 14:53:50.911018 19593 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ranger-kms
I20260619 14:53:50.911039 19593 mini_ranger_kms.cc:227] Using postgres at 127.19.34.84:39105
2026-06-19 14:53:51,083 [I] DB FLAVOR :POSTGRES
2026-06-19 14:53:51,083 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:53:51,083 [I] Checking connection
2026-06-19 14:53:51,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 14:53:51,299 [I] connection success
2026-06-19 14:53:51,300 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 14:53:51,300 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 14:53:51,300 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 14:53:51,546 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 14:53:51,546 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:53:51,546 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 14:53:51,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.34.84:39105/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 14:53:51,796 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 14:53:52.225142 19593 mini_ranger_kms.cc:310] Created kms service
I20260619 14:53:52.376896 19593 mini_ranger_kms.cc:326] Created kudu user
I20260619 14:53:52.414286 19593 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 14:53:52 dist-test-slave-3qdh krb5kdc[31359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880830, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
I20260619 14:53:52.773746 19593 mini_ranger_kms.cc:384] Added ranger policy
I20260619 14:53:52.773893 19593 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ranger-kms:/tmp/dist-test-task0Qcekt/build/debug/bin/postgresql.jar:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0Qcekt/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 14:53:52.773931 19593 mini_ranger_kms.cc:240] Using host: 127.19.34.84
I20260619 14:53:52.777761 19593 mini_ranger_kms.cc:276] Ranger KMS PID: 31752
I20260619 14:53:52.777834 19593 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.34.84:53323
Jun 19, 2026 2:53:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:53:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:53:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:53:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:53:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:53:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 14:53:54.912554 19593 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.757s user 0.000s sys 0.152s
I20260619 14:53:54.912679 19593 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 14:53:54.912773 19593 mini_ranger_kms.cc:401] 127.19.34.84:53323/kms/v1/keys
Jun 19 14:53:54 dist-test-slave-3qdh krb5kdc[31359](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781880834, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.34.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 14:53:55 dist-test-slave-3qdh krb5kdc[31359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880834, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.34.84@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
Jun 19 14:53:55 dist-test-slave-3qdh krb5kdc[31359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880834, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.34.84@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
Jun 19 14:53:57 dist-test-slave-3qdh krb5kdc[31359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:53:57.673285 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:41049
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:35227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:41049
--ranger_config_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.34.84:53323/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:53:57.813050 31803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:53:57.813364 31803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:53:57.813426 31803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:53:57.818910 31803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:53:57.818972 31803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:53:57.818998 31803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:53:57.819018 31803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:53:57.819046 31803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:53:57.825922 31803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.34.84:53323/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:41049
--ranger_config_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:41049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:53:57.826951 31803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:53:57.828191 31803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:53:57.837786 31811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:57.838006 31803 server_base.cc:1034] running on GCE node
W20260619 14:53:57.838034 31808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:57.838148 31809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:57.838958 31803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:53:57.840173 31803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:53:57.841400 31803 hybrid_clock.cc:648] HybridClock initialized: now 1781880837841383 us; error 31 us; skew 500 ppm
Jun 19 14:53:57 dist-test-slave-3qdh krb5kdc[31359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880837, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:53:57.849086 31803 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:53:57.849946 31803 webserver.cc:460] Webserver started at http://127.19.34.126:46249/ using document root <none> and password file <none>
I20260619 14:53:57.850443 31803 fs_manager.cc:359] Metadata directory not provided
I20260619 14:53:57.850492 31803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:53:57.850696 31803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:53:57 dist-test-slave-3qdh krb5kdc[31359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880837, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
I20260619 14:53:57.900831 31803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "ea4c7b23f16a4b83b25cdff378cf787f"
format_stamp: "Formatted at 2026-06-19 14:53:57 on dist-test-slave-3qdh"
server_key: "60c040090a9302e5dd2c6ccaa70740ea"
server_key_iv: "a74235dff212d4dce04e635c68f6c143"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:57.901461 31803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "ea4c7b23f16a4b83b25cdff378cf787f"
format_stamp: "Formatted at 2026-06-19 14:53:57 on dist-test-slave-3qdh"
server_key: "60c040090a9302e5dd2c6ccaa70740ea"
server_key_iv: "a74235dff212d4dce04e635c68f6c143"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:57.905141 31803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:53:57.921387 31818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:57.922617 31803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:53:57.922751 31803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "ea4c7b23f16a4b83b25cdff378cf787f"
format_stamp: "Formatted at 2026-06-19 14:53:57 on dist-test-slave-3qdh"
server_key: "60c040090a9302e5dd2c6ccaa70740ea"
server_key_iv: "a74235dff212d4dce04e635c68f6c143"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:57.922892 31803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:53:57.938951 31803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:53:57.941725 31803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:53:57.941915 31803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:53:57.949512 31803 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:41049
I20260619 14:53:57.949529 31870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:41049 every 8 connection(s)
I20260619 14:53:57.950389 31803 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:53:57.950951 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 31803
I20260619 14:53:57.951057 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:53:57.951340 19593 external_mini_cluster.cc:1428] Setting key 4aea6a2320b928cff70646e08d2d6ac0
I20260619 14:53:57.953519 31871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:53:57.960019 31871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f: Bootstrap starting.
Jun 19 14:53:57 dist-test-slave-3qdh krb5kdc[31359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:53:57.962466 31871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f: Neither blocks nor log segments found. Creating new log.
I20260619 14:53:57.963672 31871 log.cc:824] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:53:57.964996 31874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:57.952702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49568 (local address 127.19.34.126:41049)
0619 14:53:57.953106 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:57.953123 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:57.953164 (+ 41us) server_negotiation.cc:407] Connection header received
0619 14:53:57.953940 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:57.953959 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:57.954332 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:57.954697 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:57.955318 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:57.956712 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:57.957480 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:57.957982 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:57.958178 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:57.960904 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:57.960909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:57.960920 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:57.960943 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:57.962348 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:57.962863 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:57.962867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:57.962871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:57.962963 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:57.963260 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:57.963262 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:57.963263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:57.963576 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:57.963710 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:57.964003 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 14:53:57.964260 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":87,"threads_started":1}
I20260619 14:53:57.965708 31871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f: No bootstrap required, opened a new log
I20260619 14:53:57.968719 31871 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4c7b23f16a4b83b25cdff378cf787f" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41049 } }
I20260619 14:53:57.968954 31871 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:53:57.969043 31871 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea4c7b23f16a4b83b25cdff378cf787f, State: Initialized, Role: FOLLOWER
I20260619 14:53:57.969506 31871 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4c7b23f16a4b83b25cdff378cf787f" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41049 } }
I20260619 14:53:57.969645 31871 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:53:57.969722 31871 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:53:57.969812 31871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:53:57.970629 31871 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4c7b23f16a4b83b25cdff378cf787f" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41049 } }
I20260619 14:53:57.970928 31871 leader_election.cc:302] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea4c7b23f16a4b83b25cdff378cf787f; no voters:
I20260619 14:53:57.971311 31871 leader_election.cc:288] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:53:57.971398 31876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:53:57.971616 31876 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [term 1 LEADER]: Becoming Leader. State: Replica: ea4c7b23f16a4b83b25cdff378cf787f, State: Running, Role: LEADER
I20260619 14:53:57.971913 31876 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4c7b23f16a4b83b25cdff378cf787f" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41049 } }
I20260619 14:53:57.972576 31871 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:53:57.973040 31878 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea4c7b23f16a4b83b25cdff378cf787f. Latest consensus state: current_term: 1 leader_uuid: "ea4c7b23f16a4b83b25cdff378cf787f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4c7b23f16a4b83b25cdff378cf787f" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41049 } } }
I20260619 14:53:57.973178 31878 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [sys.catalog]: This master's current role is: LEADER
I20260619 14:53:57.973538 31880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:53:57.973493 31877 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea4c7b23f16a4b83b25cdff378cf787f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4c7b23f16a4b83b25cdff378cf787f" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41049 } } }
I20260619 14:53:57.973647 31877 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [sys.catalog]: This master's current role is: LEADER
I20260619 14:53:57.977000 31880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:53:57.977381 31871 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 14:53:57.986558 31880 catalog_manager.cc:1347] Generated new cluster ID: 25eee21419334134b55f2ee764a55173
I20260619 14:53:57.986624 31880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:53:58.004678 31880 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:53:58.005854 31880 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:53:58.013139 31880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f: Generated new TSK 0
I20260619 14:53:58.013707 31880 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.34.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
I20260619 14:53:58.644944 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.34.84:53323/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:41049
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:35227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:53:58.764768 31916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:53:58.765051 31916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:53:58.765110 31916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:53:58.770653 31916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:53:58.770717 31916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:53:58.770784 31916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:53:58.776216 31916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.34.84:53323/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:41049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:53:58.777217 31916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:53:58.778254 31916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:53:58.788259 31924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:58.788470 31916 server_base.cc:1034] running on GCE node
W20260619 14:53:58.788501 31923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:58.788453 31926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:58.789207 31916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:53:58.789829 31916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:53:58.791006 31916 hybrid_clock.cc:648] HybridClock initialized: now 1781880838790980 us; error 32 us; skew 500 ppm
Jun 19 14:53:58 dist-test-slave-3qdh krb5kdc[31359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880838, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:53:58.794078 31916 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:53:58.795006 31916 webserver.cc:460] Webserver started at http://127.19.34.65:41963/ using document root <none> and password file <none>
I20260619 14:53:58.795631 31916 fs_manager.cc:359] Metadata directory not provided
I20260619 14:53:58.795713 31916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:53:58.795907 31916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:53:58 dist-test-slave-3qdh krb5kdc[31359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880838, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
I20260619 14:53:58.809846 31916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "904b6156f1044e109b7a1b8841c26a73"
format_stamp: "Formatted at 2026-06-19 14:53:58 on dist-test-slave-3qdh"
server_key: "e867bd2c8527b7ebaf81fea619b62dee"
server_key_iv: "75fdaf9c22ca8f59e5755426405787e9"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:58.810583 31916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "904b6156f1044e109b7a1b8841c26a73"
format_stamp: "Formatted at 2026-06-19 14:53:58 on dist-test-slave-3qdh"
server_key: "e867bd2c8527b7ebaf81fea619b62dee"
server_key_iv: "75fdaf9c22ca8f59e5755426405787e9"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:58.840204 31916 fs_manager.cc:691] Time spent creating directory manager: real 0.029s user 0.004s sys 0.000s
I20260619 14:53:58.859434 31935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:58.860924 31916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:53:58.861020 31916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "904b6156f1044e109b7a1b8841c26a73"
format_stamp: "Formatted at 2026-06-19 14:53:58 on dist-test-slave-3qdh"
server_key: "e867bd2c8527b7ebaf81fea619b62dee"
server_key_iv: "75fdaf9c22ca8f59e5755426405787e9"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:58.861117 31916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:53:58.894850 31916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:53:58.898057 31916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:53:58.898252 31916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:53:58.898779 31916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:53:58.899682 31916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:53:58.899780 31916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:58.899889 31916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:53:58.899962 31916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:58.912914 31916 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:33927
I20260619 14:53:58.913033 32048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:33927 every 8 connection(s)
I20260619 14:53:58.913782 31916 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:53:58.915364 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 31916
I20260619 14:53:58.915488 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:53:58.915829 19593 external_mini_cluster.cc:1428] Setting key c24d9706af0d9dc185abd48c339c07c4
Jun 19 14:53:58 dist-test-slave-3qdh krb5kdc[31359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880838, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.34.66@KRBTEST.COM; defaulting to no policy
I20260619 14:53:58.931716 32052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:58.918477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:44077 (local address 127.19.34.126:41049)
0619 14:53:58.920524 (+ 2047us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:58.920536 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:58.920552 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:53:58.920594 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:58.920600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:58.920664 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:58.922136 (+ 1472us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:58.922262 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:58.923093 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:58.924576 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:58.924948 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:58.925057 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:58.927934 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:58.927937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:58.927939 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:58.927962 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:58.929793 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:58.930365 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:58.930367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:58.930368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:58.930431 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:58.930725 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:58.930729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:58.930731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:58.930963 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:58.931047 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:58.931452 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 14:53:58.931552 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1953,"thread_start_us":89,"threads_started":1}
I20260619 14:53:58.932595 32051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:58.917169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:41049 (local address 127.19.34.65:44077)
0619 14:53:58.917973 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:58.918020 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:58.918947 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:58.920996 (+ 2049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:58.921006 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:58.921351 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:53:58.921987 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:58.922006 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:58.923237 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:58.923241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:58.924415 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:58.924423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:58.925086 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:58.925090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:58.925268 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:58.925881 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:53:58.925906 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:53:58.927753 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:53:58.929918 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:58.929923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:58.929936 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:58.930255 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:58.930512 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:58.930516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:58.930518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:58.930639 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:58.931047 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:53:58.931053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:53:58.931256 (+ 203us) client_negotiation.cc:769] Sending connection context
0619 14:53:58.931417 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 14:53:58.931676 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":132,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.66@KRBTEST.COM" created.
I20260619 14:53:58.934113 32049 heartbeater.cc:347] Connected to a master server at 127.19.34.126:41049
I20260619 14:53:58.934311 32049 heartbeater.cc:464] Registering TS with master...
Jun 19 14:53:58 dist-test-slave-3qdh krb5kdc[31359](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781880838, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:53:58.934861 32049 heartbeater.cc:511] Master 127.19.34.126:41049 requested a full tablet report, sending...
I20260619 14:53:58.936390 31835 ts_manager.cc:194] Registered new tserver with Master: 904b6156f1044e109b7a1b8841c26a73 (127.19.34.65:33927)
I20260619 14:53:58.937788 31835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:44077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
I20260619 14:53:58.949704 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.34.84:53323/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:41049
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:35227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:53:59.065641 32055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:53:59.065939 32055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:53:59.065997 32055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:53:59.070644 32055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:53:59.070715 32055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:53:59.070815 32055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:53:59.075860 32055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.34.84:53323/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:41049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:53:59.076794 32055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:53:59.077790 32055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:53:59.087198 32064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:59.087538 32062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:59.087742 32061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:59.088845 32055 server_base.cc:1034] running on GCE node
I20260619 14:53:59.089228 32055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:53:59.089836 32055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:53:59.091003 32055 hybrid_clock.cc:648] HybridClock initialized: now 1781880839090972 us; error 46 us; skew 500 ppm
Jun 19 14:53:59 dist-test-slave-3qdh krb5kdc[31359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880839, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:53:59.094125 32055 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:53:59.095078 32055 webserver.cc:460] Webserver started at http://127.19.34.66:34959/ using document root <none> and password file <none>
I20260619 14:53:59.095780 32055 fs_manager.cc:359] Metadata directory not provided
I20260619 14:53:59.095899 32055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:53:59.096150 32055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:53:59 dist-test-slave-3qdh krb5kdc[31359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880839, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
I20260619 14:53:59.111310 32055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "9c1f2b80f5cf4921b7a42fc75857c0de"
format_stamp: "Formatted at 2026-06-19 14:53:59 on dist-test-slave-3qdh"
server_key: "da6280b15b9f1fabc29d118bd244bb4b"
server_key_iv: "f93a2a418596a6128d5894e00e885484"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:59.111989 32055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "9c1f2b80f5cf4921b7a42fc75857c0de"
format_stamp: "Formatted at 2026-06-19 14:53:59 on dist-test-slave-3qdh"
server_key: "da6280b15b9f1fabc29d118bd244bb4b"
server_key_iv: "f93a2a418596a6128d5894e00e885484"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:59.116302 32055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:53:59.146611 32071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:59.148222 32055 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.001s
I20260619 14:53:59.148350 32055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "9c1f2b80f5cf4921b7a42fc75857c0de"
format_stamp: "Formatted at 2026-06-19 14:53:59 on dist-test-slave-3qdh"
server_key: "da6280b15b9f1fabc29d118bd244bb4b"
server_key_iv: "f93a2a418596a6128d5894e00e885484"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:59.148470 32055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:53:59.162781 32055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:53:59.165910 32055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:53:59.166066 32055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:53:59.167359 32055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:53:59.168150 32055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:53:59.168196 32055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:59.168237 32055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:53:59.168287 32055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:59.176092 32055 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:34531
I20260619 14:53:59.176342 32184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:34531 every 8 connection(s)
I20260619 14:53:59.176951 32055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:53:59.177700 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 32055
I20260619 14:53:59.177829 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:53:59.178112 19593 external_mini_cluster.cc:1428] Setting key f048aa9b71b53581e8b73ba1f86e9161
Jun 19 14:53:59 dist-test-slave-3qdh krb5kdc[31359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880839, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.34.67@KRBTEST.COM; defaulting to no policy
I20260619 14:53:59.190505 32052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:59.178586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:50561 (local address 127.19.34.126:41049)
0619 14:53:59.178733 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:59.178738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:59.179437 (+ 699us) server_negotiation.cc:407] Connection header received
0619 14:53:59.180258 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:59.180261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:59.180329 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:59.180421 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:59.181473 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:59.182155 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:59.183103 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:59.183400 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:59.183498 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:59.186324 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:59.186329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:59.186334 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:59.186364 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:59.188134 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:59.188782 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:59.188788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:59.188790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:59.188903 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:59.189317 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:59.189320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:59.189322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:59.189510 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:59.189583 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:59.190161 (+ 578us) server_negotiation.cc:299] Negotiation successful
0619 14:53:59.190299 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:53:59.191073 32187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:59.178936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:41049 (local address 127.19.34.66:50561)
0619 14:53:59.179317 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:59.179360 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:59.180079 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:59.180563 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:59.180569 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:59.180782 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:53:59.181316 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:59.181333 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:59.182271 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:59.182274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:59.182987 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:59.182995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:59.183542 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:59.183548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:59.183750 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:59.184288 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:53:59.184332 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:53:59.186155 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:53:59.188262 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:59.188268 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:59.188282 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:59.188667 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:59.189036 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:59.189041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:59.189043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:59.189212 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:59.189660 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:53:59.189667 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:53:59.189932 (+ 265us) client_negotiation.cc:769] Sending connection context
0619 14:53:59.190086 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 14:53:59.190277 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.67@KRBTEST.COM" created.
I20260619 14:53:59.192202 32185 heartbeater.cc:347] Connected to a master server at 127.19.34.126:41049
I20260619 14:53:59.192409 32185 heartbeater.cc:464] Registering TS with master...
I20260619 14:53:59.192803 32185 heartbeater.cc:511] Master 127.19.34.126:41049 requested a full tablet report, sending...
I20260619 14:53:59.193724 31835 ts_manager.cc:194] Registered new tserver with Master: 9c1f2b80f5cf4921b7a42fc75857c0de (127.19.34.66:34531)
I20260619 14:53:59.194397 31835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:50561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
I20260619 14:53:59.209162 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.34.84:53323/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:41049
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:35227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:53:59.327368 32190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:53:59.327610 32190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:53:59.327698 32190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:53:59.331684 32190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:53:59.331754 32190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:53:59.331841 32190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:53:59.338562 32190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.34.84:53323/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:41049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:53:59.339865 32190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:53:59.341214 32190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:53:59.352043 32195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Jun 19 14:53:59 dist-test-slave-3qdh krb5kdc[31359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880838, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
W20260619 14:53:59.365100 32196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:59.365144 32198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:59.373255 32190 server_base.cc:1034] running on GCE node
I20260619 14:53:59.373807 32190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:53:59.374429 32190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:53:59.375591 32190 hybrid_clock.cc:648] HybridClock initialized: now 1781880839375553 us; error 45 us; skew 500 ppm
Jun 19 14:53:59 dist-test-slave-3qdh krb5kdc[31359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880839, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:53:59.378758 32190 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:53:59.379830 32190 webserver.cc:460] Webserver started at http://127.19.34.67:42873/ using document root <none> and password file <none>
I20260619 14:53:59.380554 32190 fs_manager.cc:359] Metadata directory not provided
I20260619 14:53:59.380779 32190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:53:59.381021 32190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:53:59 dist-test-slave-3qdh krb5kdc[31359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880839, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
I20260619 14:53:59.396073 32190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "96dbbc5c88074812b178b209ad303c74"
format_stamp: "Formatted at 2026-06-19 14:53:59 on dist-test-slave-3qdh"
server_key: "f3aa6d3ac87936c950bf2e1934607669"
server_key_iv: "e8194a2548177c84be4d651d43b3bd88"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:59.396870 32190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "96dbbc5c88074812b178b209ad303c74"
format_stamp: "Formatted at 2026-06-19 14:53:59 on dist-test-slave-3qdh"
server_key: "f3aa6d3ac87936c950bf2e1934607669"
server_key_iv: "e8194a2548177c84be4d651d43b3bd88"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:59.408551 32190 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.004s sys 0.000s
I20260619 14:53:59.424593 32205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:59.426293 32190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:53:59.426445 32190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "96dbbc5c88074812b178b209ad303c74"
format_stamp: "Formatted at 2026-06-19 14:53:59 on dist-test-slave-3qdh"
server_key: "f3aa6d3ac87936c950bf2e1934607669"
server_key_iv: "e8194a2548177c84be4d651d43b3bd88"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:59.426553 32190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:53:59.451402 32190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:53:59.454656 32190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:53:59.454833 32190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:53:59.455442 32190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:53:59.456408 32190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:53:59.456463 32190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:59.456560 32190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:53:59.456602 32190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:59.467116 32190 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:44923
I20260619 14:53:59.467132 32318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:44923 every 8 connection(s)
I20260619 14:53:59.467972 32190 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:53:59.468997 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 32190
I20260619 14:53:59.469163 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:53:59.469477 19593 external_mini_cluster.cc:1428] Setting key d9804710e2531ce37a9504331e4a5c43
Jun 19 14:53:59 dist-test-slave-3qdh krb5kdc[31359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880839, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:53:59.480206 32052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:59.471014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:60259 (local address 127.19.34.126:41049)
0619 14:53:59.471131 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:59.471136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:59.471152 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:53:59.471650 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:59.471653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:59.471703 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:59.471769 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:59.472865 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:59.473466 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:59.474702 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:59.474998 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:59.475121 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:59.477224 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:59.477228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:59.477230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:59.477256 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:59.478381 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:59.478829 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:59.478834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:59.478835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:59.478907 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:59.479270 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:59.479273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:59.479275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:59.479443 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:59.479541 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:59.479956 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 14:53:59.480056 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:53:59.480927 32321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:59.470120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:41049 (local address 127.19.34.67:60259)
0619 14:53:59.470519 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:59.470562 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:59.471450 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:59.471912 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:59.471922 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:59.472233 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:53:59.472745 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:59.472757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:59.473593 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:59.473596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:59.474541 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:59.474550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:59.475154 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:59.475158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:59.475330 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:59.475890 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:53:59.475903 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:53:59.477089 (+ 1186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:53:59.478499 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:59.478504 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:59.478515 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:59.478738 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:59.479002 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:59.479005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:59.479006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:59.479155 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:59.479543 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:53:59.479550 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:53:59.479756 (+ 206us) client_negotiation.cc:769] Sending connection context
0619 14:53:59.479920 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 14:53:59.480178 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":122,"threads_started":1}
I20260619 14:53:59.481920 32319 heartbeater.cc:347] Connected to a master server at 127.19.34.126:41049
I20260619 14:53:59.482131 32319 heartbeater.cc:464] Registering TS with master...
I20260619 14:53:59.482630 32319 heartbeater.cc:511] Master 127.19.34.126:41049 requested a full tablet report, sending...
I20260619 14:53:59.484098 31835 ts_manager.cc:194] Registered new tserver with Master: 96dbbc5c88074812b178b209ad303c74 (127.19.34.67:44923)
I20260619 14:53:59.484673 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:53:59.484959 31835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:60259
I20260619 14:53:59.485343 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 31916
I20260619 14:53:59.492970 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 32055
I20260619 14:53:59.498919 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 32190
I20260619 14:53:59.503494 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 31803
I20260619 14:53:59.508174 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:41049
--webserver_interface=127.19.34.126
--webserver_port=46249
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:35227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:41049
--ranger_config_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.34.84:53323/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.34.84:53323/kms with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:53:59.604139 32322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:53:59.604384 32322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:53:59.604451 32322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:53:59.608196 32322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:53:59.608239 32322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:53:59.608256 32322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:53:59.608270 32322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:53:59.608283 32322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:53:59.612908 32322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.34.84:53323/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:41049
--ranger_config_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:41049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=46249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:53:59.613689 32322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:53:59.614444 32322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:53:59.621366 32328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:59.621330 32327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:59.621385 32322 server_base.cc:1034] running on GCE node
W20260619 14:53:59.621348 32330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:59.621973 32322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:53:59.622803 32322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:53:59.623977 32322 hybrid_clock.cc:648] HybridClock initialized: now 1781880839623961 us; error 25 us; skew 500 ppm
Jun 19 14:53:59 dist-test-slave-3qdh krb5kdc[31359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880839, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:53:59.626607 32322 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:53:59.627308 32322 webserver.cc:460] Webserver started at http://127.19.34.126:46249/ using document root <none> and password file <none>
I20260619 14:53:59.627724 32322 fs_manager.cc:359] Metadata directory not provided
I20260619 14:53:59.627784 32322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:53:59 dist-test-slave-3qdh krb5kdc[31359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880839, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
I20260619 14:53:59.674439 32322 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:53:59.677063 32338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:59.678247 32322 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:53:59.678332 32322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "ea4c7b23f16a4b83b25cdff378cf787f"
format_stamp: "Formatted at 2026-06-19 14:53:57 on dist-test-slave-3qdh"
server_key: "60c040090a9302e5dd2c6ccaa70740ea"
server_key_iv: "a74235dff212d4dce04e635c68f6c143"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:59.678824 32322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:53:59.693900 32322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:53:59.696485 32322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:53:59.696658 32322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:53:59.703378 32322 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:41049
I20260619 14:53:59.703428 32390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:41049 every 8 connection(s)
I20260619 14:53:59.704428 32322 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:53:59.709240 32391 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:53:59.711109 32391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f: Bootstrap starting.
I20260619 14:53:59.713733 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 32322
I20260619 14:53:59.714175 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.34.84:53323/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:33927
--local_ip_for_outbound_sockets=127.19.34.65
--tserver_master_addrs=127.19.34.126:41049
--webserver_port=41963
--webserver_interface=127.19.34.65
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:35227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.34.84:53323/kms with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:53:59.727197 32391 log.cc:824] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:53:59.733174 32391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:53:59.733508 32391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f: Bootstrap complete.
I20260619 14:53:59.737313 32391 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4c7b23f16a4b83b25cdff378cf787f" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41049 } }
I20260619 14:53:59.737676 32391 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea4c7b23f16a4b83b25cdff378cf787f, State: Initialized, Role: FOLLOWER
I20260619 14:53:59.738090 32391 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4c7b23f16a4b83b25cdff378cf787f" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41049 } }
I20260619 14:53:59.738214 32391 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:53:59.738286 32391 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:53:59.738394 32391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:53:59.739979 32391 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4c7b23f16a4b83b25cdff378cf787f" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41049 } }
I20260619 14:53:59.740267 32391 leader_election.cc:302] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea4c7b23f16a4b83b25cdff378cf787f; no voters:
I20260619 14:53:59.740609 32391 leader_election.cc:288] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:53:59.740682 32395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:53:59.740900 32395 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [term 2 LEADER]: Becoming Leader. State: Replica: ea4c7b23f16a4b83b25cdff378cf787f, State: Running, Role: LEADER
I20260619 14:53:59.741166 32395 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4c7b23f16a4b83b25cdff378cf787f" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41049 } }
I20260619 14:53:59.741719 32391 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:53:59.741966 32395 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea4c7b23f16a4b83b25cdff378cf787f. Latest consensus state: current_term: 2 leader_uuid: "ea4c7b23f16a4b83b25cdff378cf787f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4c7b23f16a4b83b25cdff378cf787f" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41049 } } }
I20260619 14:53:59.742264 32395 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [sys.catalog]: This master's current role is: LEADER
I20260619 14:53:59.741954 32396 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ea4c7b23f16a4b83b25cdff378cf787f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea4c7b23f16a4b83b25cdff378cf787f" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41049 } } }
I20260619 14:53:59.742604 32396 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f [sys.catalog]: This master's current role is: LEADER
I20260619 14:53:59.743008 32400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:53:59.745980 32400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:53:59.746407 32391 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 14:53:59.746637 32400 catalog_manager.cc:1259] Loaded cluster ID: 25eee21419334134b55f2ee764a55173
I20260619 14:53:59.746722 32400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:53:59.748945 32400 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:53:59.749523 32400 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f: Loaded TSK: 0
I20260619 14:53:59.750613 32400 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:53:59.841423 32393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:53:59.841733 32393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:53:59.841827 32393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:53:59.845937 32393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:53:59.846001 32393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:53:59.846086 32393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:53:59.851518 32393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.34.84:53323/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:33927
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=41963
--tserver_master_addrs=127.19.34.126:41049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:53:59.852838 32393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:53:59.854049 32393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:53:59.864652 32431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:59.864612 32429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:53:59.865031 32428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:53:59.865000 32393 server_base.cc:1034] running on GCE node
I20260619 14:53:59.865507 32393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:53:59.866137 32393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:53:59.867309 32393 hybrid_clock.cc:648] HybridClock initialized: now 1781880839867292 us; error 32 us; skew 500 ppm
Jun 19 14:53:59 dist-test-slave-3qdh krb5kdc[31359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880839, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:53:59.870044 32393 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:53:59.870942 32393 webserver.cc:460] Webserver started at http://127.19.34.65:41963/ using document root <none> and password file <none>
I20260619 14:53:59.871448 32393 fs_manager.cc:359] Metadata directory not provided
I20260619 14:53:59.871579 32393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:53:59 dist-test-slave-3qdh krb5kdc[31359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880839, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
I20260619 14:53:59.891937 32393 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260619 14:53:59.894412 32439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:59.895802 32393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:53:59.895958 32393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "904b6156f1044e109b7a1b8841c26a73"
format_stamp: "Formatted at 2026-06-19 14:53:58 on dist-test-slave-3qdh"
server_key: "e867bd2c8527b7ebaf81fea619b62dee"
server_key_iv: "75fdaf9c22ca8f59e5755426405787e9"
server_key_version: "kuduclusterkey@0"
I20260619 14:53:59.896553 32393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:53:59.912030 32393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:53:59.915680 32393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:53:59.915925 32393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:53:59.916704 32393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:53:59.917469 32393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:53:59.917536 32393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:59.917594 32393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:53:59.917640 32393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:53:59.928020 32393 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:33927
I20260619 14:53:59.928488 32552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:33927 every 8 connection(s)
I20260619 14:53:59.929253 32393 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:53:59.930527 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 32393
I20260619 14:53:59.931020 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.34.84:53323/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:34531
--local_ip_for_outbound_sockets=127.19.34.66
--tserver_master_addrs=127.19.34.126:41049
--webserver_port=34959
--webserver_interface=127.19.34.66
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:35227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.34.84:53323/kms with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:53:59 dist-test-slave-3qdh krb5kdc[31359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880839, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:53:59.949514 32555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:59.931365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:42859 (local address 127.19.34.126:41049)
0619 14:53:59.931954 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0619 14:53:59.931975 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:53:59.932721 (+ 746us) server_negotiation.cc:407] Connection header received
0619 14:53:59.934235 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:53:59.934252 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:53:59.934656 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:53:59.934984 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:53:59.936894 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:59.938487 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:59.940054 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:53:59.940765 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:53:59.940903 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:59.943829 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:53:59.943835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:53:59.943848 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:53:59.943876 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:53:59.946093 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:59.946667 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:59.946672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:59.946677 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:59.946794 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:53:59.947165 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:53:59.947168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:53:59.947170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:53:59.947570 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:53:59.947660 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:53:59.948324 (+ 664us) server_negotiation.cc:299] Negotiation successful
0619 14:53:59.948581 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":115,"threads_started":1}
I20260619 14:53:59.950942 32556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:53:59.931500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:41049 (local address 127.19.34.65:42859)
0619 14:53:59.932542 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0619 14:53:59.932597 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:53:59.933778 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:53:59.935652 (+ 1874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:53:59.935663 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:53:59.936041 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:53:59.936685 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:59.936706 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:59.938653 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:59.938657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:59.939915 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:53:59.939926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:53:59.941164 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:53:59.941168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:53:59.941345 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:53:59.942019 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:53:59.942038 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:53:59.943635 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:53:59.946226 (+ 2591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:59.946231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:59.946243 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:59.946538 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:59.946893 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:53:59.946896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:53:59.946898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:53:59.947060 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:53:59.947734 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:53:59.947740 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:53:59.947985 (+ 245us) client_negotiation.cc:769] Sending connection context
0619 14:53:59.949848 (+ 1863us) client_negotiation.cc:240] Negotiation successful
0619 14:53:59.950174 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"thread_start_us":130,"threads_started":1}
I20260619 14:53:59.953395 32553 heartbeater.cc:347] Connected to a master server at 127.19.34.126:41049
I20260619 14:53:59.953603 32553 heartbeater.cc:464] Registering TS with master...
I20260619 14:53:59.954039 32553 heartbeater.cc:511] Master 127.19.34.126:41049 requested a full tablet report, sending...
W20260619 14:53:59.955986 32553 heartbeater.cc:644] Failed to heartbeat to 127.19.34.126:41049 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 14:53:59.956120 32553 heartbeater.cc:464] Registering TS with master...
I20260619 14:53:59.956269 32553 heartbeater.cc:511] Master 127.19.34.126:41049 requested a full tablet report, sending...
I20260619 14:53:59.957105 32553 heartbeater.cc:464] Registering TS with master...
I20260619 14:53:59.957288 32553 heartbeater.cc:511] Master 127.19.34.126:41049 requested a full tablet report, sending...
W20260619 14:53:59.958030 32553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:54:00.073678 32557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:00.073966 32557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:00.074052 32557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:00.078725 32557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:00.078783 32557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:00.078856 32557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:54:00.084813 32557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.34.84:53323/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:34531
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=34959
--tserver_master_addrs=127.19.34.126:41049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:00.085894 32557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:00.086957 32557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:00.097759 32565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:00.097882 32563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:00.097764 32562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:00.099702 32557 server_base.cc:1034] running on GCE node
I20260619 14:54:00.100028 32557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:00.100628 32557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:00.101791 32557 hybrid_clock.cc:648] HybridClock initialized: now 1781880840101771 us; error 38 us; skew 500 ppm
Jun 19 14:54:00 dist-test-slave-3qdh krb5kdc[31359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880840, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:00.104452 32557 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:54:00.105336 32557 webserver.cc:460] Webserver started at http://127.19.34.66:34959/ using document root <none> and password file <none>
I20260619 14:54:00.105921 32557 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:00.106004 32557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:54:00 dist-test-slave-3qdh krb5kdc[31359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880840, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
I20260619 14:54:00.127571 32557 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:54:00.130875 32573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:00.132257 32557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:54:00.132423 32557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "9c1f2b80f5cf4921b7a42fc75857c0de"
format_stamp: "Formatted at 2026-06-19 14:53:59 on dist-test-slave-3qdh"
server_key: "da6280b15b9f1fabc29d118bd244bb4b"
server_key_iv: "f93a2a418596a6128d5894e00e885484"
server_key_version: "kuduclusterkey@0"
I20260619 14:54:00.133051 32557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:00.150470 32557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:00.153488 32557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:00.153684 32557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:00.154292 32557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:00.155254 32557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:00.155352 32557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:00.155439 32557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:00.155498 32557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:00.164623 32557 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:34531
I20260619 14:54:00.165606 32686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:34531 every 8 connection(s)
I20260619 14:54:00.165844 32557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:54:00.170728 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 32557
I20260619 14:54:00.171283 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.34.84:53323/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:44923
--local_ip_for_outbound_sockets=127.19.34.67
--tserver_master_addrs=127.19.34.126:41049
--webserver_port=42873
--webserver_interface=127.19.34.67
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:35227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.34.84:53323/kms with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:54:00 dist-test-slave-3qdh krb5kdc[31359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880840, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:54:00.182024 32555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:00.169973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:37057 (local address 127.19.34.126:41049)
0619 14:54:00.170179 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:00.170183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:00.170197 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:54:00.170241 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:00.170244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:00.170290 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:00.170361 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:00.171534 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:00.172268 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:00.174062 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:00.174390 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:00.174521 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:00.177446 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:00.177452 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:00.177458 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:00.177488 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:00.179290 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:00.179928 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:00.179934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:00.179938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:00.180036 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:00.180608 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:00.180614 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:00.180617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:00.180845 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:00.180922 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:00.181641 (+ 719us) server_negotiation.cc:299] Negotiation successful
0619 14:54:00.181784 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260619 14:54:00.183789 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:00.167973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:41049 (local address 127.19.34.66:37057)
0619 14:54:00.169130 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:00.169167 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:00.169841 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:00.170650 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:00.170659 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:00.170970 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:00.171398 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:00.171410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:00.172664 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:00.172668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:00.173900 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:00.173912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:00.174559 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:00.174563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:00.174768 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:00.175422 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:00.175448 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:00.177271 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:00.179427 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:00.179436 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:00.179453 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:00.179801 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:00.180165 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:00.180169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:00.180174 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:00.180492 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:00.181007 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:00.181015 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:00.181321 (+ 306us) client_negotiation.cc:769] Sending connection context
0619 14:54:00.182448 (+ 1127us) client_negotiation.cc:240] Negotiation successful
0619 14:54:00.182790 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"thread_start_us":170,"threads_started":1}
I20260619 14:54:00.185286 32687 heartbeater.cc:347] Connected to a master server at 127.19.34.126:41049
I20260619 14:54:00.185488 32687 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:00.185992 32687 heartbeater.cc:511] Master 127.19.34.126:41049 requested a full tablet report, sending...
W20260619 14:54:00.187357 32687 heartbeater.cc:644] Failed to heartbeat to 127.19.34.126:41049 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 14:54:00.187474 32687 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:00.187609 32687 heartbeater.cc:511] Master 127.19.34.126:41049 requested a full tablet report, sending...
I20260619 14:54:00.188402 32687 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:00.188568 32687 heartbeater.cc:511] Master 127.19.34.126:41049 requested a full tablet report, sending...
W20260619 14:54:00.189417 32687 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:54:00.304011 32690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:00.304324 32690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:00.304422 32690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:00.309909 32690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:00.310000 32690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:00.310096 32690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:54:00.314888 32690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.34.84:53323/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:44923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=42873
--tserver_master_addrs=127.19.34.126:41049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:00.315706 32690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:00.316608 32690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:00.325284 32699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:00.325729 32697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:00.325815 32696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:00.326009 32690 server_base.cc:1034] running on GCE node
I20260619 14:54:00.326381 32690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:00.326987 32690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:00.328183 32690 hybrid_clock.cc:648] HybridClock initialized: now 1781880840328166 us; error 31 us; skew 500 ppm
Jun 19 14:54:00 dist-test-slave-3qdh krb5kdc[31359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880840, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:00.331530 32690 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:54:00.332480 32690 webserver.cc:460] Webserver started at http://127.19.34.67:42873/ using document root <none> and password file <none>
I20260619 14:54:00.333073 32690 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:00.333180 32690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:54:00 dist-test-slave-3qdh krb5kdc[31359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880840, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
I20260619 14:54:00.366346 32690 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 14:54:00.371948 32707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:00.373417 32690 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260619 14:54:00.373520 32690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "96dbbc5c88074812b178b209ad303c74"
format_stamp: "Formatted at 2026-06-19 14:53:59 on dist-test-slave-3qdh"
server_key: "f3aa6d3ac87936c950bf2e1934607669"
server_key_iv: "e8194a2548177c84be4d651d43b3bd88"
server_key_version: "kuduclusterkey@0"
I20260619 14:54:00.374172 32690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:00.390300 32690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:00.392964 32690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:00.393148 32690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:00.393728 32690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:00.394572 32690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:00.394657 32690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:00.394774 32690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:00.394804 32690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:00.405435 32690 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:44923
I20260619 14:54:00.407347 32690 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:54:00.411934 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 32690
Jun 19 14:54:00 dist-test-slave-3qdh krb5kdc[31359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880840, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:54:00.420472 352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:44923 every 8 connection(s)
I20260619 14:54:00.421331 32555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:00.410556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:57029 (local address 127.19.34.126:41049)
0619 14:54:00.410676 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:00.410682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:00.410698 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:54:00.411108 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:00.411111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:00.411164 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:00.411228 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:00.412365 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:00.413061 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:00.414579 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:00.414847 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:00.414957 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:00.417483 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:00.417487 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:00.417489 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:00.417510 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:00.419099 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:00.419635 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:00.419639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:00.419641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:00.419722 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:00.420122 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:00.420127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:00.420129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:00.420356 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:00.420434 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:00.421035 (+ 601us) server_negotiation.cc:299] Negotiation successful
0619 14:54:00.421176 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:54:00.425310 355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:00.409647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:41049 (local address 127.19.34.67:57029)
0619 14:54:00.410048 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:00.410114 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:00.410919 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:00.411373 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:00.411378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:00.411595 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:00.412167 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:00.412186 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:00.413220 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:00.413222 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:00.414463 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:00.414471 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:00.414989 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:00.414993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:00.415163 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:00.415783 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:00.415802 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:00.417351 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:00.419219 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:00.419224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:00.419235 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:00.419523 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:00.419832 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:00.419836 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:00.419838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:00.419996 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:00.420511 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:00.420516 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:00.420773 (+ 257us) client_negotiation.cc:769] Sending connection context
0619 14:54:00.424379 (+ 3606us) client_negotiation.cc:240] Negotiation successful
0619 14:54:00.424630 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":124,"threads_started":1}
I20260619 14:54:00.426545 353 heartbeater.cc:347] Connected to a master server at 127.19.34.126:41049
I20260619 14:54:00.426757 353 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:00.427222 353 heartbeater.cc:511] Master 127.19.34.126:41049 requested a full tablet report, sending...
W20260619 14:54:00.428663 353 heartbeater.cc:644] Failed to heartbeat to 127.19.34.126:41049 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 14:54:00.428768 353 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:00.428928 353 heartbeater.cc:511] Master 127.19.34.126:41049 requested a full tablet report, sending...
I20260619 14:54:00.429662 353 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:00.429833 353 heartbeater.cc:511] Master 127.19.34.126:41049 requested a full tablet report, sending...
W20260619 14:54:00.430464 353 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 14:54:00.430962 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:00.412614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49584 (local address 127.19.34.126:41049)
0619 14:54:00.412815 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:00.412821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:00.413083 (+ 262us) server_negotiation.cc:407] Connection header received
0619 14:54:00.413185 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:00.413188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:00.413242 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:00.413347 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:00.414304 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:00.415102 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:00.416361 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:00.419384 (+ 3023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:00.419487 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:00.423535 (+ 4048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:00.423538 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:00.423541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:00.423575 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:00.426263 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:00.427674 (+ 1411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:00.427677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:00.427678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:00.427764 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:00.429143 (+ 1379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:00.429149 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:00.429153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:00.429539 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:00.430153 (+ 614us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:00.430687 (+ 534us) server_negotiation.cc:299] Negotiation successful
0619 14:54:00.430775 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":73,"threads_started":1}
Jun 19 14:54:00 dist-test-slave-3qdh krb5kdc[31359](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781880840, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:00.958295 32553 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:00.958588 32553 heartbeater.cc:511] Master 127.19.34.126:41049 requested a full tablet report, sending...
I20260619 14:54:00.959714 32353 ts_manager.cc:194] Registered new tserver with Master: 904b6156f1044e109b7a1b8841c26a73 (127.19.34.65:33927)
I20260619 14:54:00.960937 32353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:42859
Jun 19 14:54:01 dist-test-slave-3qdh krb5kdc[31359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880840, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
I20260619 14:54:01.189626 32687 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:01.189874 32687 heartbeater.cc:511] Master 127.19.34.126:41049 requested a full tablet report, sending...
I20260619 14:54:01.190477 32353 ts_manager.cc:194] Registered new tserver with Master: 9c1f2b80f5cf4921b7a42fc75857c0de (127.19.34.66:34531)
I20260619 14:54:01.191040 32353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:37057
Jun 19 14:54:01 dist-test-slave-3qdh krb5kdc[31359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880840, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for HTTP/127.19.34.84@KRBTEST.COM
I20260619 14:54:01.430657 353 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:01.430920 353 heartbeater.cc:511] Master 127.19.34.126:41049 requested a full tablet report, sending...
I20260619 14:54:01.431615 32353 ts_manager.cc:194] Registered new tserver with Master: 96dbbc5c88074812b178b209ad303c74 (127.19.34.67:44923)
I20260619 14:54:01.432410 32353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:57029
I20260619 14:54:01.436018 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:54:01.452394 376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:01.441855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49588 (local address 127.19.34.126:41049)
0619 14:54:01.442385 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:01.442390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:01.442712 (+ 322us) server_negotiation.cc:407] Connection header received
0619 14:54:01.442753 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:01.442756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:01.442805 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:01.442882 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:01.443782 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.444610 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.445856 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.446219 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.446309 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:01.447982 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:01.447986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:01.447989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:01.448012 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:01.449771 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:01.450264 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:01.450268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:01.450270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:01.450353 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:01.450645 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:01.450648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:01.450650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:01.450871 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:01.450993 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:01.452053 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0619 14:54:01.452185 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":83,"threads_started":1}
I20260619 14:54:01.456338 32353 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:54:01.461004 32353 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:54:01.489113 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:01.480269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:44923 (local address 127.0.0.1:39506)
0619 14:54:01.482765 (+ 2496us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:01.482779 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:01.482869 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:01.483575 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:01.483581 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:01.483603 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:01.483736 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:01.483744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.485205 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.485209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:01.486583 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:01.486594 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.488582 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.488586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:01.488689 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:01.488731 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 14:54:01.488851 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 14:54:01.488969 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2204,"thread_start_us":103,"threads_started":1}
I20260619 14:54:01.489673 385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:01.480847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39506 (local address 127.19.34.67:44923)
0619 14:54:01.481866 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:01.481873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:01.482962 (+ 1089us) server_negotiation.cc:407] Connection header received
0619 14:54:01.483042 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:01.483047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:01.483255 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:01.483392 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:01.483871 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.485070 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.486720 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.487469 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.487578 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:01.487709 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:01.489440 (+ 1731us) server_negotiation.cc:299] Negotiation successful
0619 14:54:01.489534 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":923,"thread_start_us":104,"threads_started":1}
I20260619 14:54:01.493074 32747 tablet_service.cc:1467] Processing CreateTablet for tablet 0d220acb0fa344d5a087002b6c37403a (DEFAULT_TABLE table=test-table [id=e3293f1b4d5f46008c798bfd4aa20f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:01.494131 32747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d220acb0fa344d5a087002b6c37403a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:01.494769 384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:01.479558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:34531 (local address 127.0.0.1:34760)
0619 14:54:01.482089 (+ 2531us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:01.482134 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:01.482250 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:01.488397 (+ 6147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:01.488400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:01.488417 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:01.488512 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:01.488517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.491984 (+ 3467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.491987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:01.493210 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:01.493223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.494386 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.494389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:01.494484 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:01.494505 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:54:01.494558 (+ 53us) client_negotiation.cc:240] Negotiation successful
0619 14:54:01.494626 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2380,"thread_start_us":89,"threads_started":1}
I20260619 14:54:01.497192 383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:01.479680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34760 (local address 127.19.34.66:34531)
0619 14:54:01.487919 (+ 8239us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:01.487927 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:01.487950 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:54:01.488019 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:01.488024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:01.488221 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:01.488388 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:01.488654 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.489741 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.493361 (+ 3620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.494115 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.494212 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:01.494329 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:01.496961 (+ 2632us) server_negotiation.cc:299] Negotiation successful
0619 14:54:01.497066 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8107,"thread_start_us":85,"threads_started":1}
I20260619 14:54:01.499655 32614 tablet_service.cc:1467] Processing CreateTablet for tablet 0d220acb0fa344d5a087002b6c37403a (DEFAULT_TABLE table=test-table [id=e3293f1b4d5f46008c798bfd4aa20f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:01.500582 32614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d220acb0fa344d5a087002b6c37403a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:01.506125 388 tablet_bootstrap.cc:492] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74: Bootstrap starting.
I20260619 14:54:01.508260 388 tablet_bootstrap.cc:654] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:01.509109 388 log.cc:824] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:01.510591 384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:01.481732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:33927 (local address 127.0.0.1:33572)
0619 14:54:01.494996 (+ 13264us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:01.495008 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:01.495073 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:01.506281 (+ 11208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:01.506284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:01.506301 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:01.506364 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:01.506369 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.507753 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.507756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:01.508880 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:01.508889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.510266 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.510269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:01.510346 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:01.510367 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:54:01.510430 (+ 63us) client_negotiation.cc:240] Negotiation successful
0619 14:54:01.510484 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13213,"thread_start_us":72,"threads_started":1}
I20260619 14:54:01.511358 388 tablet_bootstrap.cc:492] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74: No bootstrap required, opened a new log
I20260619 14:54:01.511535 388 ts_tablet_manager.cc:1397] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:54:01.512811 387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:01.482083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.19.34.65:33927)
0619 14:54:01.504478 (+ 22395us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:01.504484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:01.504507 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:54:01.504585 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:01.504589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:01.504800 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:01.504946 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:01.506482 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.507633 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.509280 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.509993 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.510095 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:01.510210 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:01.512568 (+ 2358us) server_negotiation.cc:299] Negotiation successful
0619 14:54:01.512675 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22298,"thread_start_us":131,"threads_started":1}
I20260619 14:54:01.515084 388 raft_consensus.cc:348] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dbbc5c88074812b178b209ad303c74" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44923 } } peers { permanent_uuid: "9c1f2b80f5cf4921b7a42fc75857c0de" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34531 } } peers { permanent_uuid: "904b6156f1044e109b7a1b8841c26a73" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33927 } }
I20260619 14:54:01.515321 388 raft_consensus.cc:374] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:01.515363 388 raft_consensus.cc:729] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96dbbc5c88074812b178b209ad303c74, State: Initialized, Role: FOLLOWER
I20260619 14:54:01.515206 32487 tablet_service.cc:1467] Processing CreateTablet for tablet 0d220acb0fa344d5a087002b6c37403a (DEFAULT_TABLE table=test-table [id=e3293f1b4d5f46008c798bfd4aa20f14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:01.515776 388 consensus_queue.cc:260] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dbbc5c88074812b178b209ad303c74" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44923 } } peers { permanent_uuid: "9c1f2b80f5cf4921b7a42fc75857c0de" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34531 } } peers { permanent_uuid: "904b6156f1044e109b7a1b8841c26a73" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33927 } }
I20260619 14:54:01.516131 32487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d220acb0fa344d5a087002b6c37403a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:01.516359 390 tablet_bootstrap.cc:492] T 0d220acb0fa344d5a087002b6c37403a P 9c1f2b80f5cf4921b7a42fc75857c0de: Bootstrap starting.
I20260619 14:54:01.517124 388 ts_tablet_manager.cc:1428] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:54:01.517586 353 heartbeater.cc:503] Master 127.19.34.126:41049 was elected leader, sending a full tablet report...
I20260619 14:54:01.518855 390 tablet_bootstrap.cc:654] T 0d220acb0fa344d5a087002b6c37403a P 9c1f2b80f5cf4921b7a42fc75857c0de: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:01.519641 390 log.cc:824] T 0d220acb0fa344d5a087002b6c37403a P 9c1f2b80f5cf4921b7a42fc75857c0de: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:01.530431 390 tablet_bootstrap.cc:492] T 0d220acb0fa344d5a087002b6c37403a P 9c1f2b80f5cf4921b7a42fc75857c0de: No bootstrap required, opened a new log
I20260619 14:54:01.530660 390 ts_tablet_manager.cc:1397] T 0d220acb0fa344d5a087002b6c37403a P 9c1f2b80f5cf4921b7a42fc75857c0de: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260619 14:54:01.534166 390 raft_consensus.cc:348] T 0d220acb0fa344d5a087002b6c37403a P 9c1f2b80f5cf4921b7a42fc75857c0de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dbbc5c88074812b178b209ad303c74" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44923 } } peers { permanent_uuid: "9c1f2b80f5cf4921b7a42fc75857c0de" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34531 } } peers { permanent_uuid: "904b6156f1044e109b7a1b8841c26a73" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33927 } }
I20260619 14:54:01.534409 390 raft_consensus.cc:374] T 0d220acb0fa344d5a087002b6c37403a P 9c1f2b80f5cf4921b7a42fc75857c0de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:01.534482 390 raft_consensus.cc:729] T 0d220acb0fa344d5a087002b6c37403a P 9c1f2b80f5cf4921b7a42fc75857c0de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1f2b80f5cf4921b7a42fc75857c0de, State: Initialized, Role: FOLLOWER
I20260619 14:54:01.534955 390 consensus_queue.cc:260] T 0d220acb0fa344d5a087002b6c37403a P 9c1f2b80f5cf4921b7a42fc75857c0de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dbbc5c88074812b178b209ad303c74" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44923 } } peers { permanent_uuid: "9c1f2b80f5cf4921b7a42fc75857c0de" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34531 } } peers { permanent_uuid: "904b6156f1044e109b7a1b8841c26a73" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33927 } }
I20260619 14:54:01.535940 390 ts_tablet_manager.cc:1428] T 0d220acb0fa344d5a087002b6c37403a P 9c1f2b80f5cf4921b7a42fc75857c0de: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:54:01.536501 32687 heartbeater.cc:503] Master 127.19.34.126:41049 was elected leader, sending a full tablet report...
I20260619 14:54:01.536612 393 tablet_bootstrap.cc:492] T 0d220acb0fa344d5a087002b6c37403a P 904b6156f1044e109b7a1b8841c26a73: Bootstrap starting.
I20260619 14:54:01.540323 393 tablet_bootstrap.cc:654] T 0d220acb0fa344d5a087002b6c37403a P 904b6156f1044e109b7a1b8841c26a73: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:01.541108 393 log.cc:824] T 0d220acb0fa344d5a087002b6c37403a P 904b6156f1044e109b7a1b8841c26a73: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:01.543663 393 tablet_bootstrap.cc:492] T 0d220acb0fa344d5a087002b6c37403a P 904b6156f1044e109b7a1b8841c26a73: No bootstrap required, opened a new log
I20260619 14:54:01.543856 393 ts_tablet_manager.cc:1397] T 0d220acb0fa344d5a087002b6c37403a P 904b6156f1044e109b7a1b8841c26a73: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 14:54:01.548382 393 raft_consensus.cc:348] T 0d220acb0fa344d5a087002b6c37403a P 904b6156f1044e109b7a1b8841c26a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dbbc5c88074812b178b209ad303c74" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44923 } } peers { permanent_uuid: "9c1f2b80f5cf4921b7a42fc75857c0de" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34531 } } peers { permanent_uuid: "904b6156f1044e109b7a1b8841c26a73" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33927 } }
I20260619 14:54:01.548626 393 raft_consensus.cc:374] T 0d220acb0fa344d5a087002b6c37403a P 904b6156f1044e109b7a1b8841c26a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:01.548717 393 raft_consensus.cc:729] T 0d220acb0fa344d5a087002b6c37403a P 904b6156f1044e109b7a1b8841c26a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 904b6156f1044e109b7a1b8841c26a73, State: Initialized, Role: FOLLOWER
I20260619 14:54:01.549108 393 consensus_queue.cc:260] T 0d220acb0fa344d5a087002b6c37403a P 904b6156f1044e109b7a1b8841c26a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dbbc5c88074812b178b209ad303c74" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44923 } } peers { permanent_uuid: "9c1f2b80f5cf4921b7a42fc75857c0de" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34531 } } peers { permanent_uuid: "904b6156f1044e109b7a1b8841c26a73" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33927 } }
I20260619 14:54:01.549999 393 ts_tablet_manager.cc:1428] T 0d220acb0fa344d5a087002b6c37403a P 904b6156f1044e109b7a1b8841c26a73: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:54:01.550333 32553 heartbeater.cc:503] Master 127.19.34.126:41049 was elected leader, sending a full tablet report...
I20260619 14:54:01.635690 391 raft_consensus.cc:482] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:54:01.635912 391 raft_consensus.cc:504] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dbbc5c88074812b178b209ad303c74" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44923 } } peers { permanent_uuid: "9c1f2b80f5cf4921b7a42fc75857c0de" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34531 } } peers { permanent_uuid: "904b6156f1044e109b7a1b8841c26a73" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33927 } }
I20260619 14:54:01.637156 391 leader_election.cc:288] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c1f2b80f5cf4921b7a42fc75857c0de (127.19.34.66:34531), 904b6156f1044e109b7a1b8841c26a73 (127.19.34.65:33927)
I20260619 14:54:01.641206 387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:01.637726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:34271 (local address 127.19.34.65:33927)
0619 14:54:01.637825 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:01.637829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:01.637841 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:54:01.637888 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:01.637890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:01.637952 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:01.638017 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:01.638330 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.639092 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.640287 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.640676 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.640794 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:01.640841 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:01.641010 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 14:54:01.641087 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:54:01.641309 404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:01.637296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:34531 (local address 127.19.34.67:56079)
0619 14:54:01.637664 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:01.637676 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:01.637776 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:01.638116 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:01.638118 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:01.638146 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:01.638280 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:01.638288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.638989 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.638994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:01.640144 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:01.640154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.640907 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.640910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:01.641004 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:01.641023 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:54:01.641095 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 14:54:01.641155 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":29,"thread_start_us":101,"threads_started":1}
I20260619 14:54:01.641320 383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:01.637444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:56079 (local address 127.19.34.66:34531)
0619 14:54:01.637565 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:01.637569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:01.637735 (+ 166us) server_negotiation.cc:407] Connection header received
0619 14:54:01.637880 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:01.637882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:01.637922 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:01.638000 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:01.638396 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.638888 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.640267 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.640815 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.640926 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:01.640972 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:01.641108 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 14:54:01.641189 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:54:01.641698 405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:01.637368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:33927 (local address 127.19.34.67:34271)
0619 14:54:01.637664 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:01.637676 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:01.637767 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:01.638040 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:01.638043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:01.638059 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:01.638232 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:01.638239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.639204 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.639207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:01.640166 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:01.640173 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.640783 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.640786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:01.640885 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:01.640907 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:54:01.641495 (+ 588us) client_negotiation.cc:240] Negotiation successful
0619 14:54:01.641576 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260619 14:54:01.641906 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d220acb0fa344d5a087002b6c37403a" candidate_uuid: "96dbbc5c88074812b178b209ad303c74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1f2b80f5cf4921b7a42fc75857c0de" is_pre_election: true
I20260619 14:54:01.642274 32641 raft_consensus.cc:2454] T 0d220acb0fa344d5a087002b6c37403a P 9c1f2b80f5cf4921b7a42fc75857c0de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96dbbc5c88074812b178b209ad303c74 in term 0.
I20260619 14:54:01.642371 32500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d220acb0fa344d5a087002b6c37403a" candidate_uuid: "96dbbc5c88074812b178b209ad303c74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "904b6156f1044e109b7a1b8841c26a73" is_pre_election: true
I20260619 14:54:01.642639 32500 raft_consensus.cc:2454] T 0d220acb0fa344d5a087002b6c37403a P 904b6156f1044e109b7a1b8841c26a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96dbbc5c88074812b178b209ad303c74 in term 0.
I20260619 14:54:01.642764 32710 leader_election.cc:302] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96dbbc5c88074812b178b209ad303c74, 9c1f2b80f5cf4921b7a42fc75857c0de; no voters:
I20260619 14:54:01.642997 391 raft_consensus.cc:2781] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:54:01.643082 391 raft_consensus.cc:482] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:54:01.643117 391 raft_consensus.cc:3037] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:01.644363 391 raft_consensus.cc:504] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dbbc5c88074812b178b209ad303c74" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44923 } } peers { permanent_uuid: "9c1f2b80f5cf4921b7a42fc75857c0de" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34531 } } peers { permanent_uuid: "904b6156f1044e109b7a1b8841c26a73" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33927 } }
I20260619 14:54:01.644711 391 leader_election.cc:288] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 [CANDIDATE]: Term 1 election: Requested vote from peers 9c1f2b80f5cf4921b7a42fc75857c0de (127.19.34.66:34531), 904b6156f1044e109b7a1b8841c26a73 (127.19.34.65:33927)
I20260619 14:54:01.645025 32500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d220acb0fa344d5a087002b6c37403a" candidate_uuid: "96dbbc5c88074812b178b209ad303c74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "904b6156f1044e109b7a1b8841c26a73"
I20260619 14:54:01.645054 32641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d220acb0fa344d5a087002b6c37403a" candidate_uuid: "96dbbc5c88074812b178b209ad303c74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1f2b80f5cf4921b7a42fc75857c0de"
I20260619 14:54:01.645150 32500 raft_consensus.cc:3037] T 0d220acb0fa344d5a087002b6c37403a P 904b6156f1044e109b7a1b8841c26a73 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:01.645162 32641 raft_consensus.cc:3037] T 0d220acb0fa344d5a087002b6c37403a P 9c1f2b80f5cf4921b7a42fc75857c0de [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:01.646178 32500 raft_consensus.cc:2454] T 0d220acb0fa344d5a087002b6c37403a P 904b6156f1044e109b7a1b8841c26a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96dbbc5c88074812b178b209ad303c74 in term 1.
I20260619 14:54:01.646342 32641 raft_consensus.cc:2454] T 0d220acb0fa344d5a087002b6c37403a P 9c1f2b80f5cf4921b7a42fc75857c0de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96dbbc5c88074812b178b209ad303c74 in term 1.
I20260619 14:54:01.646442 32709 leader_election.cc:302] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 904b6156f1044e109b7a1b8841c26a73, 96dbbc5c88074812b178b209ad303c74; no voters:
I20260619 14:54:01.646580 391 raft_consensus.cc:2781] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:01.646773 391 raft_consensus.cc:686] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 [term 1 LEADER]: Becoming Leader. State: Replica: 96dbbc5c88074812b178b209ad303c74, State: Running, Role: LEADER
I20260619 14:54:01.647061 391 consensus_queue.cc:237] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dbbc5c88074812b178b209ad303c74" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44923 } } peers { permanent_uuid: "9c1f2b80f5cf4921b7a42fc75857c0de" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34531 } } peers { permanent_uuid: "904b6156f1044e109b7a1b8841c26a73" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33927 } }
I20260619 14:54:01.650156 32353 catalog_manager.cc:5495] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96dbbc5c88074812b178b209ad303c74 (127.19.34.67). New cstate: current_term: 1 leader_uuid: "96dbbc5c88074812b178b209ad303c74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96dbbc5c88074812b178b209ad303c74" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c1f2b80f5cf4921b7a42fc75857c0de" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "904b6156f1044e109b7a1b8841c26a73" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33927 } health_report { overall_health: UNKNOWN } } }
W20260619 14:54:01.660238 354 tablet.cc:2369] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:54:01.667761 32688 tablet.cc:2369] T 0d220acb0fa344d5a087002b6c37403a P 9c1f2b80f5cf4921b7a42fc75857c0de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:54:01.679684 385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:01.676131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39518 (local address 127.19.34.67:44923)
0619 14:54:01.676226 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:01.676229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:01.676240 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:54:01.676411 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:01.676413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:01.676458 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:01.676533 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:54:01.676784 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.677221 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.678307 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:01.678514 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:01.678627 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:01.678915 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:54:01.679282 (+ 367us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:54:01.679388 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:01.679496 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 14:54:01.679547 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260619 14:54:01.681478 32554 tablet.cc:2369] T 0d220acb0fa344d5a087002b6c37403a P 904b6156f1044e109b7a1b8841c26a73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:54:01.684823 32500 raft_consensus.cc:1261] T 0d220acb0fa344d5a087002b6c37403a P 904b6156f1044e109b7a1b8841c26a73 [term 1 FOLLOWER]: Refusing update from remote peer 96dbbc5c88074812b178b209ad303c74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:01.684933 32641 raft_consensus.cc:1261] T 0d220acb0fa344d5a087002b6c37403a P 9c1f2b80f5cf4921b7a42fc75857c0de [term 1 FOLLOWER]: Refusing update from remote peer 96dbbc5c88074812b178b209ad303c74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:01.685483 391 consensus_queue.cc:1035] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "904b6156f1044e109b7a1b8841c26a73" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:01.685865 406 consensus_queue.cc:1035] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1f2b80f5cf4921b7a42fc75857c0de" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:01.703312 32355 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:01.703449 32355 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:01.708740 32355 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea4c7b23f16a4b83b25cdff378cf787f: Sending DeleteTablet for 3 replicas of tablet 0d220acb0fa344d5a087002b6c37403a
I20260619 14:54:01.709481 32487 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d220acb0fa344d5a087002b6c37403a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:01 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:33572
I20260619 14:54:01.709625 32614 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d220acb0fa344d5a087002b6c37403a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:01 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:34760
I20260619 14:54:01.710180 32747 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d220acb0fa344d5a087002b6c37403a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:01 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:39506
I20260619 14:54:01.710484 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 32393
I20260619 14:54:01.716488 421 tablet_replica.cc:330] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74: stopping tablet replica
I20260619 14:54:01.720453 420 tablet_replica.cc:330] T 0d220acb0fa344d5a087002b6c37403a P 9c1f2b80f5cf4921b7a42fc75857c0de: stopping tablet replica
I20260619 14:54:01.720595 421 raft_consensus.cc:2229] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:54:01.720726 420 raft_consensus.cc:2229] T 0d220acb0fa344d5a087002b6c37403a P 9c1f2b80f5cf4921b7a42fc75857c0de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:54:01.720944 421 raft_consensus.cc:2258] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:01.720992 420 raft_consensus.cc:2258] T 0d220acb0fa344d5a087002b6c37403a P 9c1f2b80f5cf4921b7a42fc75857c0de [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 14:54:01.721365 32340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.34.65:33927 (error 108)
I20260619 14:54:01.722275 421 ts_tablet_manager.cc:1905] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:01.722297 420 ts_tablet_manager.cc:1905] T 0d220acb0fa344d5a087002b6c37403a P 9c1f2b80f5cf4921b7a42fc75857c0de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:01.722532 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 32557
I20260619 14:54:01.722606 384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:01.722327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:33927 (local address 127.0.0.1:33586)
0619 14:54:01.722450 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:01.722518 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.65:33927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 14:54:01.725584 421 ts_tablet_manager.cc:1918] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:01.725679 421 log.cc:1196] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/0d220acb0fa344d5a087002b6c37403a
I20260619 14:54:01.725996 421 ts_tablet_manager.cc:1939] T 0d220acb0fa344d5a087002b6c37403a P 96dbbc5c88074812b178b209ad303c74: Deleting consensus metadata
W20260619 14:54:01.726773 32340 catalog_manager.cc:4584] TS 904b6156f1044e109b7a1b8841c26a73 (127.19.34.65:33927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d220acb0fa344d5a087002b6c37403a: Network error: Client connection negotiation failed: client connection to 127.19.34.65:33927: connect: Connection refused (error 111)
I20260619 14:54:01.727018 32340 catalog_manager.cc:4841] TS 96dbbc5c88074812b178b209ad303c74 (127.19.34.67:44923): tablet 0d220acb0fa344d5a087002b6c37403a (table test-table [id=e3293f1b4d5f46008c798bfd4aa20f14]) successfully deleted
I20260619 14:54:01.731922 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 32690
I20260619 14:54:01.732007 384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:01.731691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:34531 (local address 127.0.0.1:34770)
0619 14:54:01.731908 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:01.731946 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.66:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260619 14:54:01.736939 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 32322
I20260619 14:54:01.743060 19593 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 14:54:02.077288 19593 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 14:54:02.077399 19593 mini_ranger.cc:66] Stopping Ranger...
I20260619 14:54:02.422305 19593 mini_ranger.cc:68] Stopped Ranger
2026-06-19 14:54:02.422 UTC [31389] LOG: received smart shutdown request
2026-06-19 14:54:02.425 UTC [31389] LOG: background worker "logical replication launcher" (PID 31398) exited with exit code 1
2026-06-19 14:54:02.425 UTC [31393] LOG: shutting down
2026-06-19 14:54:02.560 UTC [31389] LOG: database system is shut down
2026-06-19T14:54:02Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (42819 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:54:02 dist-test-slave-3qdh krb5kdc[431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:54:02 dist-test-slave-3qdh krb5kdc[431](info): set up 2 sockets
Jun 19 14:54:02 dist-test-slave-3qdh krb5kdc[431](info): commencing operation
krb5kdc: starting...
W20260619 14:54:03.636029 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:54:03 dist-test-slave-3qdh krb5kdc[431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:54:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:54:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:54:03.741009 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:41061
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:45803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:41061,127.19.34.125:40129,127.19.34.124:34807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:03.834988 444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:03.835230 444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:03.835294 444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:03.839005 444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:03.839082 444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:03.839115 444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:03.839165 444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:03.839200 444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:03.843432 444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:45803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:41061,127.19.34.125:40129,127.19.34.124:34807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:41061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:03.844213 444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:03.845043 444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:03.852380 450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:03.852530 449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:03.852501 452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:03.852504 444 server_base.cc:1034] running on GCE node
I20260619 14:54:03.853123 444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:03.853986 444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:03.855160 444 hybrid_clock.cc:648] HybridClock initialized: now 1781880843855136 us; error 41 us; skew 500 ppm
Jun 19 14:54:03 dist-test-slave-3qdh krb5kdc[431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880843, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:03.866446 444 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:54:03.867166 444 webserver.cc:460] Webserver started at http://127.19.34.126:43765/ using document root <none> and password file <none>
I20260619 14:54:03.867597 444 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:03.867692 444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:03.867889 444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:03.869571 444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "32bca08a6e5b45ffbe6272b73306498b"
format_stamp: "Formatted at 2026-06-19 14:54:03 on dist-test-slave-3qdh"
server_key: "98c709c1ce965fe6226db027606a9a4e"
server_key_iv: "dbf32f5b24a41d4d785af3e869f03656"
server_key_version: "encryptionkey@0"
I20260619 14:54:03.870038 444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "32bca08a6e5b45ffbe6272b73306498b"
format_stamp: "Formatted at 2026-06-19 14:54:03 on dist-test-slave-3qdh"
server_key: "98c709c1ce965fe6226db027606a9a4e"
server_key_iv: "dbf32f5b24a41d4d785af3e869f03656"
server_key_version: "encryptionkey@0"
I20260619 14:54:03.873024 444 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:54:03.874908 459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:03.875998 444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:54:03.876130 444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "32bca08a6e5b45ffbe6272b73306498b"
format_stamp: "Formatted at 2026-06-19 14:54:03 on dist-test-slave-3qdh"
server_key: "98c709c1ce965fe6226db027606a9a4e"
server_key_iv: "dbf32f5b24a41d4d785af3e869f03656"
server_key_version: "encryptionkey@0"
I20260619 14:54:03.876246 444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:03.889616 444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:03.891783 444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:03.891979 444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:03.898048 444 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:41061
I20260619 14:54:03.898061 511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:41061 every 8 connection(s)
I20260619 14:54:03.898768 444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:03.900995 512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:03.903857 512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:54:03.905087 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 444
I20260619 14:54:03.905211 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:54:03.905467 19593 external_mini_cluster.cc:1428] Setting key b2ed23ebe4bc75cc08479a0d4a40b064
WARNING: no policy specified for kudu/127.19.34.125@KRBTEST.COM; defaulting to no policy
Jun 19 14:54:03 dist-test-slave-3qdh krb5kdc[431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880843, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.125@KRBTEST.COM" created.
I20260619 14:54:03.917567 515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:03.906447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34828 (local address 127.19.34.126:41061)
0619 14:54:03.906725 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:03.906733 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:03.906762 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:54:03.907480 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:03.907487 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:03.907693 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:03.907905 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:03.908803 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:03.909667 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:03.911024 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:03.911333 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:03.911425 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:03.913559 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:03.913564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:03.913572 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:03.913595 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:03.914820 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:03.915315 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:03.915324 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:03.915326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:03.915443 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:03.915792 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:03.915798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:03.915803 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:03.916287 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:03.916441 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:03.916820 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 14:54:03.916956 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":107,"threads_started":1}
I20260619 14:54:03.917574 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:03.905762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:41061 (local address 127.0.0.1:34828)
0619 14:54:03.906239 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:03.906287 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:03.907217 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:03.907918 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:03.907922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:03.908334 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:03.908672 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:03.908690 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:03.909777 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:03.909783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:03.910923 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:03.910933 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:03.911463 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:03.911469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:03.911582 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:03.912044 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:03.912061 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:03.913440 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:03.914912 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:03.914920 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:03.914927 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:03.915209 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:03.915569 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:03.915572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:03.915573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:03.915693 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:03.916418 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:03.916422 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:03.916589 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 14:54:03.916775 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 14:54:03.917035 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260619 14:54:03.919629 512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:54:03.920352 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:03.920014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:40129 (local address 127.0.0.1:34826)
0619 14:54:03.920163 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:03.920257 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.125:40129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260619 14:54:03.920555 461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.34.125:40129: connect: Connection refused (error 111)
I20260619 14:54:03.921275 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:03.921060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:40129 (local address 127.0.0.1:34832)
0619 14:54:03.921165 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:03.921217 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.125:40129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 14:54:03.921418 512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.125:40129: Network error: Client connection negotiation failed: client connection to 127.19.34.125:40129: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.125.keytab.
Entry for principal kudu/127.19.34.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.125.keytab.
I20260619 14:54:03.929241 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.125:40129
--webserver_interface=127.19.34.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.125.keytab
--principal=kudu/127.19.34.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:45803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:41061,127.19.34.125:40129,127.19.34.124:34807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:54:03.970677 512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } attempt: 1
I20260619 14:54:03.971467 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:03.971169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:40129 (local address 127.0.0.1:34846)
0619 14:54:03.971327 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:03.971387 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.125:40129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 14:54:03.972054 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:03.971867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:40129 (local address 127.0.0.1:34856)
0619 14:54:03.971953 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:03.971997 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.125:40129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 14:54:03.972193 512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.125:40129: Network error: Client connection negotiation failed: client connection to 127.19.34.125:40129: connect: Connection refused (error 111)
W20260619 14:54:04.026906 519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:04.027130 519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:04.027168 519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:04.031049 519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:04.031100 519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:04.031118 519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:04.031168 519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:04.031203 519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:04.035979 519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:45803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:41061,127.19.34.125:40129,127.19.34.124:34807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.125.keytab
--principal=kudu/127.19.34.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.125:40129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.34.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:04.036834 519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:04.037688 519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:54:04.040347 512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } attempt: 2
I20260619 14:54:04.041128 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.040797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:40129 (local address 127.0.0.1:34864)
0619 14:54:04.040967 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.041030 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.125:40129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 14:54:04.041836 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.041623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:40129 (local address 127.0.0.1:34876)
0619 14:54:04.041741 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.041784 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.125:40129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 14:54:04.041951 512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.125:40129: Network error: Client connection negotiation failed: client connection to 127.19.34.125:40129: connect: Connection refused (error 111)
W20260619 14:54:04.045220 524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:04.045190 527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:04.045212 525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:04.045634 519 server_base.cc:1034] running on GCE node
I20260619 14:54:04.046103 519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:04.046988 519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:04.048162 519 hybrid_clock.cc:648] HybridClock initialized: now 1781880844048151 us; error 25 us; skew 500 ppm
Jun 19 14:54:04 dist-test-slave-3qdh krb5kdc[431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880844, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:04.050729 519 init.cc:377] Logged in from keytab as kudu/127.19.34.125@KRBTEST.COM (short username kudu)
I20260619 14:54:04.051478 519 webserver.cc:460] Webserver started at http://127.19.34.125:39183/ using document root <none> and password file <none>
I20260619 14:54:04.051952 519 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:04.052032 519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:04.052227 519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:04.054010 519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-1/data/instance:
uuid: "5c13df262237411cb6b3a2ef17bbf681"
format_stamp: "Formatted at 2026-06-19 14:54:04 on dist-test-slave-3qdh"
server_key: "845c7d1577e221378178c823d19e445b"
server_key_iv: "5cff7244dcc098f1ae81f373a3c8f561"
server_key_version: "encryptionkey@0"
I20260619 14:54:04.054530 519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-1/wal/instance:
uuid: "5c13df262237411cb6b3a2ef17bbf681"
format_stamp: "Formatted at 2026-06-19 14:54:04 on dist-test-slave-3qdh"
server_key: "845c7d1577e221378178c823d19e445b"
server_key_iv: "5cff7244dcc098f1ae81f373a3c8f561"
server_key_version: "encryptionkey@0"
I20260619 14:54:04.057898 519 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:54:04.060119 534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:04.061194 519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:54:04.061313 519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-1/wal
uuid: "5c13df262237411cb6b3a2ef17bbf681"
format_stamp: "Formatted at 2026-06-19 14:54:04 on dist-test-slave-3qdh"
server_key: "845c7d1577e221378178c823d19e445b"
server_key_iv: "5cff7244dcc098f1ae81f373a3c8f561"
server_key_version: "encryptionkey@0"
I20260619 14:54:04.061430 519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:04.069514 519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:04.071717 519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:04.071874 519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:04.077898 519 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.125:40129
I20260619 14:54:04.077911 586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.125:40129 every 8 connection(s)
I20260619 14:54:04.078632 519 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-1/data/info.pb
I20260619 14:54:04.081135 587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:04.083774 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 519
I20260619 14:54:04.083860 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-1/wal/instance
I20260619 14:54:04.084086 19593 external_mini_cluster.cc:1428] Setting key ae76573f5dc80b1dab52e209fbb46e71
I20260619 14:54:04.084206 587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.34.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.124@KRBTEST.COM" created.
Jun 19 14:54:04 dist-test-slave-3qdh krb5kdc[431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880844, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.125@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:54:04.100049 515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.085875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34834 (local address 127.19.34.126:41061)
0619 14:54:04.086014 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:04.086018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:04.086786 (+ 768us) server_negotiation.cc:407] Connection header received
0619 14:54:04.088009 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:04.088011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:04.088056 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:04.088131 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:04.089627 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.090092 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.091631 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.091840 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.091909 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.095583 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:04.095587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:04.095589 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:04.095614 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:04.098027 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.098535 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.098540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.098542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.098644 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.098963 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.098968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.098969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.099182 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:04.099279 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:04.099761 (+ 482us) server_negotiation.cc:299] Negotiation successful
0619 14:54:04.099881 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:54:04.100838 590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.086222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:41061 (local address 127.0.0.1:34834)
0619 14:54:04.086640 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.086688 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:04.087789 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:04.088338 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:04.088348 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:04.088812 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:04.089478 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:04.089498 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.090207 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.090211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:04.091518 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:04.091532 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.091975 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.091982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:04.092201 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.092669 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:04.092683 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:04.095388 (+ 2705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:04.098194 (+ 2806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:04.098198 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:04.098211 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:04.098436 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:04.098736 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:04.098741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:04.098743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:04.098871 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:04.099287 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:04.099296 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:04.099523 (+ 227us) client_negotiation.cc:769] Sending connection context
0619 14:54:04.099704 (+ 181us) client_negotiation.cc:240] Negotiation successful
0619 14:54:04.099998 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":84,"threads_started":1}
I20260619 14:54:04.102238 587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:54:04 dist-test-slave-3qdh krb5kdc[431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880844, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.125@KRBTEST.COM for kudu/127.19.34.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.124.keytab.
Entry for principal kudu/127.19.34.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.124.keytab.
I20260619 14:54:04.109328 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.124:34807
--webserver_interface=127.19.34.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.124.keytab
--principal=kudu/127.19.34.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:45803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:41061,127.19.34.125:40129,127.19.34.124:34807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:54:04.110833 590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.102707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:40129 (local address 127.0.0.1:34878)
0619 14:54:04.102788 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.102803 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:04.102866 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:04.103376 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:04.103379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:04.103524 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:04.103638 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:04.103643 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.104601 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.104604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:04.105219 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:04.105226 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.105775 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.105778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:04.105879 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.106306 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:04.106321 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:04.107630 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:04.109392 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:04.109397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:04.109401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:04.109650 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:04.109897 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:04.109902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:04.109906 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:04.109997 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:04.110365 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:04.110368 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:04.110436 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 14:54:04.110536 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 14:54:04.110653 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":3}
I20260619 14:54:04.110848 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.102796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34878 (local address 127.19.34.125:40129)
0619 14:54:04.103049 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:04.103057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:04.103071 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:54:04.103104 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:04.103107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:04.103278 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:04.103397 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:04.103729 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.104495 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.105331 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.105665 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.105776 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.107762 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:04.107770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:04.107780 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:04.107810 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:04.109294 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.109746 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.109750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.109751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.109817 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.110078 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.110079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.110080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.110276 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:04.110395 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:04.110602 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 14:54:04.110702 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":101,"threads_started":1}
I20260619 14:54:04.112641 587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:54:04.113293 590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.113059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:34807 (local address 127.0.0.1:44696)
0619 14:54:04.113161 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.113232 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:34807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 14:54:04.113466 535 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.34.124:34807: connect: Connection refused (error 111)
I20260619 14:54:04.114203 590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.114014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:34807 (local address 127.0.0.1:44702)
0619 14:54:04.114108 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.114152 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:34807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 14:54:04.114360 587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.124:34807: Network error: Client connection negotiation failed: client connection to 127.19.34.124:34807: connect: Connection refused (error 111)
I20260619 14:54:04.133095 512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } attempt: 3
Jun 19 14:54:04 dist-test-slave-3qdh krb5kdc[431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880843, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for kudu/127.19.34.125@KRBTEST.COM
I20260619 14:54:04.141036 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.133629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34894 (local address 127.19.34.125:40129)
0619 14:54:04.133741 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:04.133747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:04.133761 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:54:04.133899 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:04.133902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:04.133944 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:04.134015 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:04.134400 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.134869 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.135652 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.135903 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.136009 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.138178 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:04.138186 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:04.138189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:04.138209 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:04.139486 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.139871 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.139875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.139877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.139950 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.140215 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.140220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.140221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.140449 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:04.140546 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:04.140784 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 14:54:04.140889 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:54:04.141036 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.133560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:40129 (local address 127.0.0.1:34894)
0619 14:54:04.133695 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.133709 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:04.133804 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:04.134032 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:04.134035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:04.134207 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:04.134298 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:04.134303 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.134996 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.135001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:04.135537 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:04.135545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.136013 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.136016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:04.136131 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.136612 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:04.136628 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:04.138040 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:04.139586 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:04.139590 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:04.139593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:04.139776 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:04.140034 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:04.140038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:04.140040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:04.140124 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:04.140536 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:04.140539 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:04.140636 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 14:54:04.140737 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 14:54:04.140871 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 14:54:04.142508 512 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:54:04.143177 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.142959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:34807 (local address 127.0.0.1:44712)
0619 14:54:04.143064 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.143111 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:34807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 14:54:04.143802 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.143609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:34807 (local address 127.0.0.1:44718)
0619 14:54:04.143730 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.143764 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:34807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260619 14:54:04.143911 512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.124:34807: Network error: Client connection negotiation failed: client connection to 127.19.34.124:34807: connect: Connection refused (error 111)
I20260619 14:54:04.163623 587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } attempt: 1
I20260619 14:54:04.164216 590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.163988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:34807 (local address 127.0.0.1:44726)
0619 14:54:04.164099 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.164157 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:34807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 14:54:04.165007 590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.164794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:34807 (local address 127.0.0.1:44732)
0619 14:54:04.164872 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.164928 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:34807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260619 14:54:04.165166 587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.124:34807: Network error: Client connection negotiation failed: client connection to 127.19.34.124:34807: connect: Connection refused (error 111)
I20260619 14:54:04.175058 512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } attempt: 1
I20260619 14:54:04.175664 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.175435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:34807 (local address 127.0.0.1:44738)
0619 14:54:04.175544 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.175600 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:34807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 14:54:04.176378 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.176122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:34807 (local address 127.0.0.1:44754)
0619 14:54:04.176243 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.176283 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:34807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 14:54:04.176512 512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.124:34807: Network error: Client connection negotiation failed: client connection to 127.19.34.124:34807: connect: Connection refused (error 111)
W20260619 14:54:04.213734 593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:04.213949 593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:04.214025 593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:04.217799 593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:04.217851 593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:04.217871 593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:04.217888 593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:04.217905 593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:04.222697 593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:45803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:41061,127.19.34.125:40129,127.19.34.124:34807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.124.keytab
--principal=kudu/127.19.34.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.124:34807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.34.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:04.223541 593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:04.224479 593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:04.232180 599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:04.232234 600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:04.232180 602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:04.232282 593 server_base.cc:1034] running on GCE node
I20260619 14:54:04.232879 593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:04.233350 587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } attempt: 2
I20260619 14:54:04.233783 593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:04.234026 590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.233748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:34807 (local address 127.0.0.1:44764)
0619 14:54:04.233917 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.233960 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:34807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260619 14:54:04.234562 590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.234404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:34807 (local address 127.0.0.1:44776)
0619 14:54:04.234480 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.234517 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:34807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260619 14:54:04.234685 587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.124:34807: Network error: Client connection negotiation failed: client connection to 127.19.34.124:34807: connect: Connection refused (error 111)
I20260619 14:54:04.234953 593 hybrid_clock.cc:648] HybridClock initialized: now 1781880844234919 us; error 42 us; skew 500 ppm
Jun 19 14:54:04 dist-test-slave-3qdh krb5kdc[431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880844, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:04.237466 593 init.cc:377] Logged in from keytab as kudu/127.19.34.124@KRBTEST.COM (short username kudu)
I20260619 14:54:04.238210 593 webserver.cc:460] Webserver started at http://127.19.34.124:35535/ using document root <none> and password file <none>
I20260619 14:54:04.238711 593 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:04.238792 593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:04.238986 593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:04.240708 593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-2/data/instance:
uuid: "69ac44af13a640b5ba80053bee470879"
format_stamp: "Formatted at 2026-06-19 14:54:04 on dist-test-slave-3qdh"
server_key: "20e7f9223c72b38e3b9c7170d0511c43"
server_key_iv: "a2f28bf69240ab5de8dc2283ece162fd"
server_key_version: "encryptionkey@0"
I20260619 14:54:04.241171 593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-2/wal/instance:
uuid: "69ac44af13a640b5ba80053bee470879"
format_stamp: "Formatted at 2026-06-19 14:54:04 on dist-test-slave-3qdh"
server_key: "20e7f9223c72b38e3b9c7170d0511c43"
server_key_iv: "a2f28bf69240ab5de8dc2283ece162fd"
server_key_version: "encryptionkey@0"
I20260619 14:54:04.244416 593 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:54:04.246508 609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:04.247506 593 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:54:04.247633 593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-2/wal
uuid: "69ac44af13a640b5ba80053bee470879"
format_stamp: "Formatted at 2026-06-19 14:54:04 on dist-test-slave-3qdh"
server_key: "20e7f9223c72b38e3b9c7170d0511c43"
server_key_iv: "a2f28bf69240ab5de8dc2283ece162fd"
server_key_version: "encryptionkey@0"
I20260619 14:54:04.247726 593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:04.251667 512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } attempt: 2
I20260619 14:54:04.252357 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.252071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:34807 (local address 127.0.0.1:44786)
0619 14:54:04.252195 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.252247 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:34807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 14:54:04.252970 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.252789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:34807 (local address 127.0.0.1:44788)
0619 14:54:04.252876 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.252914 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:34807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 14:54:04.253080 512 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.124:34807: Network error: Client connection negotiation failed: client connection to 127.19.34.124:34807: connect: Connection refused (error 111)
I20260619 14:54:04.266628 593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:04.269125 593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:04.269304 593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:04.275554 593 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.124:34807
I20260619 14:54:04.275579 661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.124:34807 every 8 connection(s)
I20260619 14:54:04.276516 593 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-2/data/info.pb
I20260619 14:54:04.278900 662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:04.282264 662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:54:04.285104 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 593
I20260619 14:54:04.285214 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880697016847-19593-0/minicluster-data/master-2/wal/instance
I20260619 14:54:04.285481 19593 external_mini_cluster.cc:1428] Setting key 0acdd308165899a411b65b5afa7b3669
Jun 19 14:54:04 dist-test-slave-3qdh krb5kdc[431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880844, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.124@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
Jun 19 14:54:04 dist-test-slave-3qdh krb5kdc[431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.124@KRBTEST.COM
I20260619 14:54:04.294869 515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.283931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34838 (local address 127.19.34.126:41061)
0619 14:54:04.284044 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:04.284047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:04.284789 (+ 742us) server_negotiation.cc:407] Connection header received
0619 14:54:04.285927 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:04.285930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:04.285985 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:04.286085 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:04.287210 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.287742 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.288808 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.289014 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.289115 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.291543 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:04.291546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:04.291549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:04.291573 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:04.292937 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.293485 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.293488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.293489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.293555 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.293876 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.293878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.293879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.294018 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:04.294080 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:04.294594 (+ 514us) server_negotiation.cc:299] Negotiation successful
0619 14:54:04.294714 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:54:04.295698 666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.286254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44798 (local address 127.19.34.124:34807)
0619 14:54:04.286629 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:04.286640 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:04.286665 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:54:04.286730 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:04.286736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:04.286884 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:04.287031 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:04.287817 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.288740 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.289512 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.289842 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.289954 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.292468 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:04.292475 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:04.292487 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:04.292514 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:04.294116 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.294458 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.294465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.294470 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.294545 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.294804 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.294807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.294808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.295055 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:04.295234 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:04.295401 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 14:54:04.295522 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":244,"thread_start_us":131,"threads_started":1}
I20260619 14:54:04.295706 664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.284267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:41061 (local address 127.0.0.1:34838)
0619 14:54:04.284669 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.284705 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:04.285719 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:04.286270 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:04.286279 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:04.286671 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:04.287072 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:04.287084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.287862 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.287867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:04.288686 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:04.288695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.289150 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.289154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:04.289302 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.289791 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:04.289806 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:04.291425 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:04.293036 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:04.293046 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:04.293057 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:04.293390 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:04.293635 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:04.293641 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:04.293644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:04.293789 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:04.294102 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:04.294108 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:04.294349 (+ 241us) client_negotiation.cc:769] Sending connection context
0619 14:54:04.294520 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 14:54:04.294776 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":99,"threads_started":1}
I20260619 14:54:04.296844 662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:54:04 dist-test-slave-3qdh krb5kdc[431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880844, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.124@KRBTEST.COM for kudu/127.19.34.125@KRBTEST.COM
Jun 19 14:54:04 dist-test-slave-3qdh krb5kdc[431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:54:04.304271 664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.297230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:40129 (local address 127.0.0.1:34902)
0619 14:54:04.297359 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.297368 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:04.297416 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:04.297631 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:04.297633 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:04.297813 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:04.297911 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:04.297917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.298639 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.298643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:04.299151 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:04.299158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.299588 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.299591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:04.299680 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.300069 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:04.300080 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:04.301369 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:04.302983 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:04.302989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:04.302991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:04.303196 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:04.303466 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:04.303469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:04.303470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:04.303542 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:04.303909 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:04.303911 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:04.303973 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 14:54:04.304039 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 14:54:04.304116 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 14:54:04.304417 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.297308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34902 (local address 127.19.34.125:40129)
0619 14:54:04.297427 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:04.297433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:04.297447 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:54:04.297488 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:04.297490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:04.297531 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:04.297600 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:04.298031 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.298530 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.299279 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.299497 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.299591 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.301480 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:04.301482 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:04.301484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:04.301504 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:04.302893 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.303280 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.303286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.303288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.303367 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.303620 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.303623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.303625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.303826 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:04.303922 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:04.304120 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 14:54:04.304228 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:54:04.305047 662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:54:04.305766 515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.297907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34842 (local address 127.19.34.126:41061)
0619 14:54:04.298020 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:04.298026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:04.298039 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:54:04.298201 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:04.298204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:04.298247 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:04.298321 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:04.298857 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.299316 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.299973 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.300163 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.300255 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.302628 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:04.302631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:04.302632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:04.302656 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:04.304104 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.304587 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.304607 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.304609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.304694 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.304983 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.304987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.304989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.305208 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:04.305431 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:04.305508 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 14:54:04.305618 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 14:54:04 dist-test-slave-3qdh krb5kdc[431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880844, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.124@KRBTEST.COM for kudu/127.19.34.124@KRBTEST.COM
Jun 19 14:54:04 dist-test-slave-3qdh krb5kdc[431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.125@KRBTEST.COM
I20260619 14:54:04.313064 664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.305433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:34807 (local address 127.0.0.1:44804)
0619 14:54:04.305542 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.305557 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:04.305623 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:04.306153 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:04.306155 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:04.306277 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:04.306365 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:04.306371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.307230 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.307232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:04.307973 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:04.307981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.308524 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.308527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:04.308605 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.308967 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:04.308979 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:04.310278 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:04.311817 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:04.311823 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:04.311826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:04.312004 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:04.312231 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:04.312235 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:04.312237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:04.312346 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:04.312636 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:04.312639 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:04.312705 (+ 66us) client_negotiation.cc:769] Sending connection context
0619 14:54:04.312796 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 14:54:04.312908 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 14:54:04.313107 666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.305619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44804 (local address 127.19.34.124:34807)
0619 14:54:04.305958 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:04.305962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:04.305974 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:54:04.306005 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:04.306007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:04.306053 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:04.306139 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:04.306464 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.307132 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.308068 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.308430 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.308523 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.310390 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:04.310392 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:04.310394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:04.310413 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:04.311725 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.312083 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.312088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.312090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.312156 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.312415 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.312417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.312419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.312553 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:04.312636 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:04.312848 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 14:54:04.312958 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":276}
I20260619 14:54:04.314713 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.306757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34916 (local address 127.19.34.125:40129)
0619 14:54:04.306848 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:04.306852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:04.306886 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:54:04.306977 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:04.306979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:04.307007 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:04.307060 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:04.307777 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.308390 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.309404 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.309766 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.309868 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.311934 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:04.311937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:04.311938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:04.311957 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:04.313298 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.313694 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.313697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.313698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.313781 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.314050 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.314052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.314054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.314240 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:04.314444 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:04.314507 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 14:54:04.314594 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:54:04.316676 662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879: Bootstrap starting.
I20260619 14:54:04.317610 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:54:04.319155 662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:04.319967 662 log.cc:824] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:04.321777 662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879: No bootstrap required, opened a new log
I20260619 14:54:04.325096 662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } } peers { permanent_uuid: "5c13df262237411cb6b3a2ef17bbf681" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } } peers { permanent_uuid: "69ac44af13a640b5ba80053bee470879" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } }
I20260619 14:54:04.325325 662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:04.325387 662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ac44af13a640b5ba80053bee470879, State: Initialized, Role: FOLLOWER
I20260619 14:54:04.325874 662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } } peers { permanent_uuid: "5c13df262237411cb6b3a2ef17bbf681" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } } peers { permanent_uuid: "69ac44af13a640b5ba80053bee470879" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } }
I20260619 14:54:04.326045 587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } attempt: 3
I20260619 14:54:04.327167 662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:04.328344 675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } } peers { permanent_uuid: "5c13df262237411cb6b3a2ef17bbf681" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } } peers { permanent_uuid: "69ac44af13a640b5ba80053bee470879" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } } }
I20260619 14:54:04.328466 675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:54:04.330258 666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.319704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44806 (local address 127.19.34.124:34807)
0619 14:54:04.319888 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:04.319893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:04.320405 (+ 512us) server_negotiation.cc:407] Connection header received
0619 14:54:04.320542 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:04.320545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:04.320580 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:04.320624 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:04.321549 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.322200 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.323441 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.323754 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.323851 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.325326 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:04.325329 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:04.325333 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:04.325361 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:04.327550 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.329020 (+ 1470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.329023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.329025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.329102 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.329370 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.329373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.329374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.329597 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:04.329691 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:04.330021 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 14:54:04.330117 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
Jun 19 14:54:04 dist-test-slave-3qdh krb5kdc[431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880844, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.125@KRBTEST.COM for kudu/127.19.34.124@KRBTEST.COM
I20260619 14:54:04.333957 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.319878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34924 (local address 127.19.34.125:40129)
0619 14:54:04.319960 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:04.319963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:04.320344 (+ 381us) server_negotiation.cc:407] Connection header received
0619 14:54:04.320512 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:04.320517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:04.320557 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:04.320602 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:04.321549 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.322205 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.323947 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.324261 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.324398 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.326794 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:04.326797 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:04.326800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:04.326823 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:04.328685 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.330935 (+ 2250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.330940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.330941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.331021 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.331340 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.331344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.331345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.331537 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:04.331637 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:04.331984 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 14:54:04.332099 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:54:04.330950 593 master_runner.cc:186] Error getting master registration for 127.19.34.126:41061: OK, instance_id { permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" instance_seqno: 1781880843896267 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:54:04.334237 593 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 14:54:04.336067 687 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:54:04.336131 687 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:54:04.336884 515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.319792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34846 (local address 127.19.34.126:41061)
0619 14:54:04.319960 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:04.319964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:04.320530 (+ 566us) server_negotiation.cc:407] Connection header received
0619 14:54:04.324870 (+ 4340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:04.324874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:04.324937 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:04.325010 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:04.326693 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.327987 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.329168 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.329462 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.329547 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.331754 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:04.331757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:04.331772 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:04.331812 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:04.333517 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.334628 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.334631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.334632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.334712 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.335975 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.335979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.335981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.336199 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:04.336287 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:04.336645 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 14:54:04.336750 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260619 14:54:04.337752 590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.326364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:34807 (local address 127.0.0.1:44814)
0619 14:54:04.326544 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.326557 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:04.326619 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:04.327994 (+ 1375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:04.327997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:04.328442 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:04.328547 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:04.328556 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.329521 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.329524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:04.330230 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:04.330243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.331611 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.331615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:04.331699 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.332108 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:04.332121 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:04.333445 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:04.335910 (+ 2465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:04.335914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:04.335918 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:04.336156 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:04.336558 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:04.336560 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:04.336563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:04.336643 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:04.337337 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:04.337339 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:04.337406 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 14:54:04.337502 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 14:54:04.337617 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260619 14:54:04.337821 676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.326735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44814 (local address 127.19.34.124:34807)
0619 14:54:04.327752 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:04.327759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:04.327770 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:54:04.327818 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:04.327821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:04.327857 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:04.328080 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:04.328655 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.329391 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.330349 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.330633 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.330722 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.333597 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:04.333601 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:04.333604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:04.333626 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:04.335792 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.336378 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.336381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.336382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.336457 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.336743 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.336748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.336752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.336913 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:04.337021 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:04.337571 (+ 550us) server_negotiation.cc:299] Negotiation successful
0619 14:54:04.337660 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":963,"thread_start_us":107,"threads_started":1}
I20260619 14:54:04.340065 587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681: Bootstrap starting.
I20260619 14:54:04.342208 587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:04.343241 587 log.cc:824] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:04.344942 587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681: No bootstrap required, opened a new log
I20260619 14:54:04.347561 587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } } peers { permanent_uuid: "5c13df262237411cb6b3a2ef17bbf681" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } } peers { permanent_uuid: "69ac44af13a640b5ba80053bee470879" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } }
I20260619 14:54:04.347704 587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:04.347769 587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c13df262237411cb6b3a2ef17bbf681, State: Initialized, Role: FOLLOWER
I20260619 14:54:04.348127 587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } } peers { permanent_uuid: "5c13df262237411cb6b3a2ef17bbf681" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } } peers { permanent_uuid: "69ac44af13a640b5ba80053bee470879" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } }
I20260619 14:54:04.348557 690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } } peers { permanent_uuid: "5c13df262237411cb6b3a2ef17bbf681" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } } peers { permanent_uuid: "69ac44af13a640b5ba80053bee470879" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } } }
I20260619 14:54:04.348667 690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:54:04.349188 587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:04.352247 512 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } attempt: 3
W20260619 14:54:04.355679 701 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:54:04.355751 701 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:54:04.352540 519 master_runner.cc:186] Error getting master registration for 127.19.34.126:41061: OK, instance_id { permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" instance_seqno: 1781880843896267 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:54:04.356235 519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 19 14:54:04 dist-test-slave-3qdh krb5kdc[431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880843, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for kudu/127.19.34.124@KRBTEST.COM
I20260619 14:54:04.362594 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.352581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:34807 (local address 127.0.0.1:44828)
0619 14:54:04.352685 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:04.352699 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:04.352775 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:04.353178 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:04.353181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:04.353359 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:04.353453 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:04.353461 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.355261 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.355264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:04.356019 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:04.356029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.356631 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.356641 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:04.356728 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.357200 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:04.357214 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:04.358538 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:04.360271 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:04.360274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:04.360276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:04.360517 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:04.361673 (+ 1156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:04.361676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:04.361678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:04.361775 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:04.362151 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:04.362154 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:04.362238 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 14:54:04.362346 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 14:54:04.362452 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 14:54:04.362615 676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:04.352760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44828 (local address 127.19.34.124:34807)
0619 14:54:04.352899 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:04.352902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:04.352912 (+ 10us) server_negotiation.cc:407] Connection header received
0619 14:54:04.352941 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:04.352944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:04.352984 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:04.353070 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:04.353580 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.354258 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.356153 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:04.356513 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:04.356615 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:04.358668 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:04.358672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:04.358674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:04.358698 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:04.360175 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.360625 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.360629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.360630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.360719 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:04.361865 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:04.361868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:04.361870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:04.362049 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:04.362139 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:04.362401 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 14:54:04.362495 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 14:54:04.365779 512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b: Bootstrap starting.
I20260619 14:54:04.368129 512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:04.369629 512 log.cc:824] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:04.371599 512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b: No bootstrap required, opened a new log
I20260619 14:54:04.375263 512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } } peers { permanent_uuid: "5c13df262237411cb6b3a2ef17bbf681" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } } peers { permanent_uuid: "69ac44af13a640b5ba80053bee470879" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } }
I20260619 14:54:04.375474 512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:04.375525 512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32bca08a6e5b45ffbe6272b73306498b, State: Initialized, Role: FOLLOWER
I20260619 14:54:04.375985 512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } } peers { permanent_uuid: "5c13df262237411cb6b3a2ef17bbf681" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } } peers { permanent_uuid: "69ac44af13a640b5ba80053bee470879" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } }
I20260619 14:54:04.376426 704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } } peers { permanent_uuid: "5c13df262237411cb6b3a2ef17bbf681" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } } peers { permanent_uuid: "69ac44af13a640b5ba80053bee470879" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } } }
I20260619 14:54:04.376595 704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:54:04.377203 512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:54:04.382141 715 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:54:04.382248 715 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:54:04.385574 444 master_runner.cc:263] No leader master found from master 32bca08a6e5b45ffbe6272b73306498b
I20260619 14:54:04.385640 444 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 14:54:04.393184 465 tablet.cc:2369] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:54:04.520440 615 tablet.cc:2369] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:54:04.524189 690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:54:04.524362 690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } } peers { permanent_uuid: "5c13df262237411cb6b3a2ef17bbf681" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } } peers { permanent_uuid: "69ac44af13a640b5ba80053bee470879" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } }
I20260619 14:54:04.525182 690 leader_election.cc:288] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32bca08a6e5b45ffbe6272b73306498b (127.19.34.126:41061), 69ac44af13a640b5ba80053bee470879 (127.19.34.124:34807)
I20260619 14:54:04.525691 636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c13df262237411cb6b3a2ef17bbf681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ac44af13a640b5ba80053bee470879" is_pre_election: true
I20260619 14:54:04.525849 486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c13df262237411cb6b3a2ef17bbf681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32bca08a6e5b45ffbe6272b73306498b" is_pre_election: true
I20260619 14:54:04.526010 636 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c13df262237411cb6b3a2ef17bbf681 in term 0.
I20260619 14:54:04.526103 486 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c13df262237411cb6b3a2ef17bbf681 in term 0.
I20260619 14:54:04.526413 535 leader_election.cc:302] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c13df262237411cb6b3a2ef17bbf681, 69ac44af13a640b5ba80053bee470879; no voters:
I20260619 14:54:04.526688 690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:54:04.526784 690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:54:04.526831 690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:04.528232 690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } } peers { permanent_uuid: "5c13df262237411cb6b3a2ef17bbf681" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } } peers { permanent_uuid: "69ac44af13a640b5ba80053bee470879" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } }
I20260619 14:54:04.528600 690 leader_election.cc:288] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [CANDIDATE]: Term 1 election: Requested vote from peers 32bca08a6e5b45ffbe6272b73306498b (127.19.34.126:41061), 69ac44af13a640b5ba80053bee470879 (127.19.34.124:34807)
I20260619 14:54:04.528930 636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c13df262237411cb6b3a2ef17bbf681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ac44af13a640b5ba80053bee470879"
I20260619 14:54:04.528998 486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5c13df262237411cb6b3a2ef17bbf681" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32bca08a6e5b45ffbe6272b73306498b"
I20260619 14:54:04.529065 636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:04.529106 486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:04.529978 636 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c13df262237411cb6b3a2ef17bbf681 in term 1.
I20260619 14:54:04.530228 535 leader_election.cc:302] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c13df262237411cb6b3a2ef17bbf681, 69ac44af13a640b5ba80053bee470879; no voters:
I20260619 14:54:04.530316 486 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c13df262237411cb6b3a2ef17bbf681 in term 1.
I20260619 14:54:04.530431 690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:04.530629 690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [term 1 LEADER]: Becoming Leader. State: Replica: 5c13df262237411cb6b3a2ef17bbf681, State: Running, Role: LEADER
I20260619 14:54:04.530920 690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } } peers { permanent_uuid: "5c13df262237411cb6b3a2ef17bbf681" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } } peers { permanent_uuid: "69ac44af13a640b5ba80053bee470879" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } }
I20260619 14:54:04.532720 717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c13df262237411cb6b3a2ef17bbf681. Latest consensus state: current_term: 1 leader_uuid: "5c13df262237411cb6b3a2ef17bbf681" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } } peers { permanent_uuid: "5c13df262237411cb6b3a2ef17bbf681" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } } peers { permanent_uuid: "69ac44af13a640b5ba80053bee470879" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } } }
I20260619 14:54:04.532819 717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:04.533494 719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:04.534569 719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:04.539198 636 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879 [term 1 FOLLOWER]: Refusing update from remote peer 5c13df262237411cb6b3a2ef17bbf681: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:04.539317 486 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b [term 1 FOLLOWER]: Refusing update from remote peer 5c13df262237411cb6b3a2ef17bbf681: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:04.539829 690 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ac44af13a640b5ba80053bee470879" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:04.539947 717 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:04.543578 675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c13df262237411cb6b3a2ef17bbf681. Latest consensus state: current_term: 1 leader_uuid: "5c13df262237411cb6b3a2ef17bbf681" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } } peers { permanent_uuid: "5c13df262237411cb6b3a2ef17bbf681" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } } peers { permanent_uuid: "69ac44af13a640b5ba80053bee470879" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } } }
I20260619 14:54:04.543694 675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:54:04.543632 704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c13df262237411cb6b3a2ef17bbf681. Latest consensus state: current_term: 1 leader_uuid: "5c13df262237411cb6b3a2ef17bbf681" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } } peers { permanent_uuid: "5c13df262237411cb6b3a2ef17bbf681" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } } peers { permanent_uuid: "69ac44af13a640b5ba80053bee470879" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } } }
I20260619 14:54:04.543730 704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:54:04.546288 717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c13df262237411cb6b3a2ef17bbf681" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } } peers { permanent_uuid: "5c13df262237411cb6b3a2ef17bbf681" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } } peers { permanent_uuid: "69ac44af13a640b5ba80053bee470879" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } } }
I20260619 14:54:04.546388 717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:04.546491 717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5c13df262237411cb6b3a2ef17bbf681" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } } peers { permanent_uuid: "5c13df262237411cb6b3a2ef17bbf681" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } } peers { permanent_uuid: "69ac44af13a640b5ba80053bee470879" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } } }
I20260619 14:54:04.546475 675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c13df262237411cb6b3a2ef17bbf681" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } } peers { permanent_uuid: "5c13df262237411cb6b3a2ef17bbf681" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } } peers { permanent_uuid: "69ac44af13a640b5ba80053bee470879" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } } }
I20260619 14:54:04.546553 717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:04.546558 675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:54:04.546887 721 mvcc.cc:205] Tried to move back new op lower bound from 7298583939225772032 to 7298583939200958464. Current Snapshot: MvccSnapshot[applied={T|T < 7298583939225772032}]
I20260619 14:54:04.547753 704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5c13df262237411cb6b3a2ef17bbf681" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32bca08a6e5b45ffbe6272b73306498b" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41061 } } peers { permanent_uuid: "5c13df262237411cb6b3a2ef17bbf681" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 40129 } } peers { permanent_uuid: "69ac44af13a640b5ba80053bee470879" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 34807 } } }
I20260619 14:54:04.547888 704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:54:04.549594 719 catalog_manager.cc:1347] Generated new cluster ID: f6a84609f92148d787586e56a4918930
I20260619 14:54:04.549661 719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:04.567082 719 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:54:04.568053 719 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:04.575019 719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c13df262237411cb6b3a2ef17bbf681: Generated new TSK 0
I20260619 14:54:04.575686 719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:54:05.337370 687 catalog_manager.cc:1259] Loaded cluster ID: f6a84609f92148d787586e56a4918930
I20260619 14:54:05.337450 687 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879: loading cluster ID for follower catalog manager: success
I20260619 14:54:05.338975 687 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879: acquiring CA information for follower catalog manager: success
I20260619 14:54:05.339988 687 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 69ac44af13a640b5ba80053bee470879: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:54:05.383483 715 catalog_manager.cc:1259] Loaded cluster ID: f6a84609f92148d787586e56a4918930
I20260619 14:54:05.383572 715 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b: loading cluster ID for follower catalog manager: success
I20260619 14:54:05.385166 715 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b: acquiring CA information for follower catalog manager: success
I20260619 14:54:05.386204 715 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 32bca08a6e5b45ffbe6272b73306498b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:54:05.626760 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 444
I20260619 14:54:05.632704 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 519
I20260619 14:54:05.638198 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 593
2026-06-19T14:54:05Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2971 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:54:05 dist-test-slave-3qdh krb5kdc[734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:54:05 dist-test-slave-3qdh krb5kdc[734](info): set up 2 sockets
Jun 19 14:54:05 dist-test-slave-3qdh krb5kdc[734](info): commencing operation
krb5kdc: starting...
W20260619 14:54:06.608129 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:54:06 dist-test-slave-3qdh krb5kdc[734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:54:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:54:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:54:06.713250 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:45029
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:40681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:45029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:06.812808 747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:06.813056 747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:06.813162 747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:06.817866 747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:06.817924 747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:06.817956 747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:06.817989 747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:06.818022 747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:06.823376 747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:40681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:45029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:45029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:06.824188 747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:06.825042 747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:06.832337 752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:06.832418 755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:06.832329 753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:06.832472 747 server_base.cc:1034] running on GCE node
I20260619 14:54:06.833053 747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:06.833945 747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:06.835126 747 hybrid_clock.cc:648] HybridClock initialized: now 1781880846835117 us; error 30 us; skew 500 ppm
Jun 19 14:54:06 dist-test-slave-3qdh krb5kdc[734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880846, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:06.837594 747 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:54:06.838331 747 webserver.cc:460] Webserver started at http://127.19.34.126:34939/ using document root <none> and password file <none>
I20260619 14:54:06.838754 747 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:06.838804 747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:06.839012 747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:06.840730 747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "fd38428a5dae48e99a0a47b85de50f9d"
format_stamp: "Formatted at 2026-06-19 14:54:06 on dist-test-slave-3qdh"
server_key: "e929408da67fae012bbe0a3322cf43cf"
server_key_iv: "39ef61e5e822b7681ae59dc0be59fd81"
server_key_version: "encryptionkey@0"
I20260619 14:54:06.841203 747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "fd38428a5dae48e99a0a47b85de50f9d"
format_stamp: "Formatted at 2026-06-19 14:54:06 on dist-test-slave-3qdh"
server_key: "e929408da67fae012bbe0a3322cf43cf"
server_key_iv: "39ef61e5e822b7681ae59dc0be59fd81"
server_key_version: "encryptionkey@0"
I20260619 14:54:06.844290 747 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:54:06.846369 762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:06.847373 747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:54:06.847505 747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "fd38428a5dae48e99a0a47b85de50f9d"
format_stamp: "Formatted at 2026-06-19 14:54:06 on dist-test-slave-3qdh"
server_key: "e929408da67fae012bbe0a3322cf43cf"
server_key_iv: "39ef61e5e822b7681ae59dc0be59fd81"
server_key_version: "encryptionkey@0"
I20260619 14:54:06.847591 747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:06.866650 747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:06.868870 747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:06.868992 747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:06.874773 747 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:45029
I20260619 14:54:06.874794 814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:45029 every 8 connection(s)
I20260619 14:54:06.875558 747 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:06.877862 815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:06.878003 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 747
I20260619 14:54:06.878120 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:54:06.878350 19593 external_mini_cluster.cc:1428] Setting key c3036aa78c55842b0194201908e569e5
I20260619 14:54:06.883935 815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d: Bootstrap starting.
I20260619 14:54:06.886198 815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d: Neither blocks nor log segments found. Creating new log.
Jun 19 14:54:06 dist-test-slave-3qdh krb5kdc[734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:54:06.887059 815 log.cc:824] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:06.888876 815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d: No bootstrap required, opened a new log
I20260619 14:54:06.891124 818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:06.879523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38972 (local address 127.19.34.126:45029)
0619 14:54:06.879894 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:06.879910 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:06.879946 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:54:06.880713 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:06.880733 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:06.881086 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:06.881413 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:06.882048 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:06.883376 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:06.884141 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:06.884732 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:06.884886 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:06.887369 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:06.887373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:06.887383 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:06.887428 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:06.888899 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:06.889291 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:06.889300 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:06.889303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:06.889407 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:06.889650 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:06.889653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:06.889655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:06.889945 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:06.890103 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:06.890352 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 14:54:06.890566 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":109,"threads_started":1}
I20260619 14:54:06.891927 815 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd38428a5dae48e99a0a47b85de50f9d" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45029 } }
I20260619 14:54:06.892148 815 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:06.892208 815 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd38428a5dae48e99a0a47b85de50f9d, State: Initialized, Role: FOLLOWER
I20260619 14:54:06.892612 815 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd38428a5dae48e99a0a47b85de50f9d" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45029 } }
I20260619 14:54:06.892765 815 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:06.892827 815 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:06.892922 815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:06.893493 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:54:06.893851 815 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd38428a5dae48e99a0a47b85de50f9d" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45029 } }
I20260619 14:54:06.894160 815 leader_election.cc:302] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd38428a5dae48e99a0a47b85de50f9d; no voters:
I20260619 14:54:06.894462 815 leader_election.cc:288] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:54:06.894558 820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:06.894742 820 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d [term 1 LEADER]: Becoming Leader. State: Replica: fd38428a5dae48e99a0a47b85de50f9d, State: Running, Role: LEADER
I20260619 14:54:06.895002 820 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd38428a5dae48e99a0a47b85de50f9d" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45029 } }
I20260619 14:54:06.895526 815 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:06.895985 820 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd38428a5dae48e99a0a47b85de50f9d. Latest consensus state: current_term: 1 leader_uuid: "fd38428a5dae48e99a0a47b85de50f9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd38428a5dae48e99a0a47b85de50f9d" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45029 } } }
I20260619 14:54:06.896162 820 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:06.896759 821 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd38428a5dae48e99a0a47b85de50f9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd38428a5dae48e99a0a47b85de50f9d" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45029 } } }
I20260619 14:54:06.896835 821 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:06.896868 826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:06.899613 826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:06.904318 826 catalog_manager.cc:1347] Generated new cluster ID: 583a953efaf64fb18d019825fcd78bb4
I20260619 14:54:06.904371 826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:06.936657 826 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:54:06.937704 826 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:06.948414 826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd38428a5dae48e99a0a47b85de50f9d: Generated new TSK 0
I20260619 14:54:06.949057 826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:54:06.999625 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 747
2026-06-19T14:54:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1359 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:54:07 dist-test-slave-3qdh krb5kdc[842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:54:07 dist-test-slave-3qdh krb5kdc[842](info): set up 2 sockets
Jun 19 14:54:07 dist-test-slave-3qdh krb5kdc[842](info): commencing operation
krb5kdc: starting...
W20260619 14:54:07.968856 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:54:08 dist-test-slave-3qdh krb5kdc[842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:54:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:54:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:54:08.072358 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:45351
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:45559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:45351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:08.168223 855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:08.168510 855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:08.168586 855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:08.172212 855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:08.172291 855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:08.172346 855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:08.172403 855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:08.172443 855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:08.177141 855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:45559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:45351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:45351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:08.177973 855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:08.178807 855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:08.186177 863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:08.186185 861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:08.186190 860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:08.186383 855 server_base.cc:1034] running on GCE node
I20260619 14:54:08.186930 855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:08.187716 855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:08.188855 855 hybrid_clock.cc:648] HybridClock initialized: now 1781880848188840 us; error 30 us; skew 500 ppm
Jun 19 14:54:08 dist-test-slave-3qdh krb5kdc[842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880848, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:08.191124 855 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:54:08.191815 855 webserver.cc:460] Webserver started at http://127.19.34.126:38623/ using document root <none> and password file <none>
I20260619 14:54:08.192265 855 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:08.192360 855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:08.192556 855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:08.194202 855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "5fda20c76804435e88c1573a73fc0ab5"
format_stamp: "Formatted at 2026-06-19 14:54:08 on dist-test-slave-3qdh"
server_key: "da42ef64eca699ce16385e16099e1951"
server_key_iv: "fe4227969ffa66f60a48443485ab6221"
server_key_version: "encryptionkey@0"
I20260619 14:54:08.194689 855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "5fda20c76804435e88c1573a73fc0ab5"
format_stamp: "Formatted at 2026-06-19 14:54:08 on dist-test-slave-3qdh"
server_key: "da42ef64eca699ce16385e16099e1951"
server_key_iv: "fe4227969ffa66f60a48443485ab6221"
server_key_version: "encryptionkey@0"
I20260619 14:54:08.198089 855 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 14:54:08.200007 870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:08.200974 855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:54:08.201084 855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "5fda20c76804435e88c1573a73fc0ab5"
format_stamp: "Formatted at 2026-06-19 14:54:08 on dist-test-slave-3qdh"
server_key: "da42ef64eca699ce16385e16099e1951"
server_key_iv: "fe4227969ffa66f60a48443485ab6221"
server_key_version: "encryptionkey@0"
I20260619 14:54:08.201187 855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:08.220928 855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:08.223662 855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:08.223835 855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:08.230139 855 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:45351
I20260619 14:54:08.230386 922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:45351 every 8 connection(s)
I20260619 14:54:08.230974 855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:08.233337 923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:08.237074 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 855
I20260619 14:54:08.237182 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:54:08.237392 19593 external_mini_cluster.cc:1428] Setting key f068c54ec68cb3e43c12743c23b4337b
I20260619 14:54:08.238073 923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5: Bootstrap starting.
I20260619 14:54:08.240337 923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:08.241096 923 log.cc:824] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:08.242777 923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5: No bootstrap required, opened a new log
Jun 19 14:54:08 dist-test-slave-3qdh krb5kdc[842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:54:08.245986 923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fda20c76804435e88c1573a73fc0ab5" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45351 } }
I20260619 14:54:08.246258 923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:08.246337 923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fda20c76804435e88c1573a73fc0ab5, State: Initialized, Role: FOLLOWER
I20260619 14:54:08.246753 923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fda20c76804435e88c1573a73fc0ab5" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45351 } }
I20260619 14:54:08.246876 923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:08.246933 923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:08.247011 923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:08.247735 923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fda20c76804435e88c1573a73fc0ab5" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45351 } }
I20260619 14:54:08.248031 923 leader_election.cc:302] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fda20c76804435e88c1573a73fc0ab5; no voters:
I20260619 14:54:08.248375 923 leader_election.cc:288] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:54:08.248474 928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:08.248672 928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 1 LEADER]: Becoming Leader. State: Replica: 5fda20c76804435e88c1573a73fc0ab5, State: Running, Role: LEADER
I20260619 14:54:08.248977 928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fda20c76804435e88c1573a73fc0ab5" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45351 } }
I20260619 14:54:08.249511 923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:08.250295 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:08.238445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45746 (local address 127.19.34.126:45351)
0619 14:54:08.238791 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:08.238806 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:08.238839 (+ 33us) server_negotiation.cc:407] Connection header received
0619 14:54:08.239531 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:08.239549 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:08.239872 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:08.240192 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:08.240899 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.242166 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.242944 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.243354 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.243509 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:08.245651 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:08.245656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:08.245666 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:08.245690 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:08.247410 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:08.247795 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:08.247807 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:08.247812 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:08.247959 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:08.248311 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:08.248318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:08.248322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:08.248781 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:08.248959 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:08.249236 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 14:54:08.249490 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":101,"threads_started":1}
I20260619 14:54:08.250427 929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fda20c76804435e88c1573a73fc0ab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fda20c76804435e88c1573a73fc0ab5" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45351 } } }
I20260619 14:54:08.250563 929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:08.250749 929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fda20c76804435e88c1573a73fc0ab5. Latest consensus state: current_term: 1 leader_uuid: "5fda20c76804435e88c1573a73fc0ab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fda20c76804435e88c1573a73fc0ab5" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45351 } } }
I20260619 14:54:08.250808 929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [sys.catalog]: This master's current role is: LEADER
W20260619 14:54:08.253580 943 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:54:08.253660 943 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:54:08.253707 937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:08.254390 937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:08.259089 937 catalog_manager.cc:1347] Generated new cluster ID: 3683abfa882f484c8883630a35127119
I20260619 14:54:08.259145 937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:08.284514 937 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:54:08.285456 937 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:08.289139 937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5: Generated new TSK 0
I20260619 14:54:08.289676 937 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.34.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
I20260619 14:54:08.318384 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:45351
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:45559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:08.413579 949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:08.413758 949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:08.413791 949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:08.417354 949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:08.417395 949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:08.417440 949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:54:08.421744 949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:45559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:45351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:08.422513 949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:08.423247 949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:08.431283 954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:08.431318 957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:08.431311 955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:08.431365 949 server_base.cc:1034] running on GCE node
I20260619 14:54:08.431828 949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:08.432384 949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:08.433534 949 hybrid_clock.cc:648] HybridClock initialized: now 1781880848433522 us; error 27 us; skew 500 ppm
Jun 19 14:54:08 dist-test-slave-3qdh krb5kdc[842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880848, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:08.435864 949 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:54:08.436645 949 webserver.cc:460] Webserver started at http://127.19.34.65:34327/ using document root <none> and password file <none>
I20260619 14:54:08.437085 949 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:08.437160 949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:08.437345 949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:08.438937 949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "bd5de741a6004d7cb5132cbe2e86365d"
format_stamp: "Formatted at 2026-06-19 14:54:08 on dist-test-slave-3qdh"
server_key: "bb42d1c75047b6999106dd06f2602935"
server_key_iv: "3f28884ec4939ad766f43d8358741d5c"
server_key_version: "encryptionkey@0"
I20260619 14:54:08.439401 949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "bd5de741a6004d7cb5132cbe2e86365d"
format_stamp: "Formatted at 2026-06-19 14:54:08 on dist-test-slave-3qdh"
server_key: "bb42d1c75047b6999106dd06f2602935"
server_key_iv: "3f28884ec4939ad766f43d8358741d5c"
server_key_version: "encryptionkey@0"
I20260619 14:54:08.442464 949 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 14:54:08.444283 964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:08.445232 949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:54:08.445340 949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "bd5de741a6004d7cb5132cbe2e86365d"
format_stamp: "Formatted at 2026-06-19 14:54:08 on dist-test-slave-3qdh"
server_key: "bb42d1c75047b6999106dd06f2602935"
server_key_iv: "3f28884ec4939ad766f43d8358741d5c"
server_key_version: "encryptionkey@0"
I20260619 14:54:08.445451 949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:08.467177 949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:08.469730 949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:08.469889 949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:08.470372 949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:08.471092 949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:08.471164 949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:08.471230 949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:08.471267 949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:08.479413 949 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:44209
I20260619 14:54:08.479470 1077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:44209 every 8 connection(s)
I20260619 14:54:08.480175 949 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:54:08.482915 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 949
I20260619 14:54:08.483006 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:54:08.483281 19593 external_mini_cluster.cc:1428] Setting key 9168fbed7a6d9cb3bb2cf72cd84a031f
Jun 19 14:54:08 dist-test-slave-3qdh krb5kdc[842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880848, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.34.66@KRBTEST.COM; defaulting to no policy
I20260619 14:54:08.492733 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:08.481762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:51299 (local address 127.19.34.126:45351)
0619 14:54:08.481878 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:08.481887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:08.482554 (+ 667us) server_negotiation.cc:407] Connection header received
0619 14:54:08.483950 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:08.483954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:08.484003 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:08.484050 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:08.485406 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.485928 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.486993 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.487212 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.487311 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:08.489527 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:08.489530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:08.489531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:08.489551 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:08.490801 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:08.491270 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:08.491274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:08.491276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:08.491347 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:08.491690 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:08.491693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:08.491695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:08.491852 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:08.491949 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:08.492461 (+ 512us) server_negotiation.cc:299] Negotiation successful
0619 14:54:08.492573 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:54:08.493395 1080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:08.482017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.65:51299)
0619 14:54:08.482418 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:08.482454 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:08.483720 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:08.484409 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:08.484417 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:08.484811 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:08.485250 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:08.485263 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.486057 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.486060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:08.486884 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:08.486891 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.487341 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.487344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:08.487484 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:08.488028 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:08.488042 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:08.489384 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:08.490907 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:08.490912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:08.490923 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:08.491182 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:08.491425 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:08.491429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:08.491431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:08.491586 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:08.491948 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:08.491954 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:08.492189 (+ 235us) client_negotiation.cc:769] Sending connection context
0619 14:54:08.492418 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 14:54:08.492702 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":102,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.66@KRBTEST.COM" created.
I20260619 14:54:08.494362 1078 heartbeater.cc:347] Connected to a master server at 127.19.34.126:45351
I20260619 14:54:08.494544 1078 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:08.494923 1078 heartbeater.cc:511] Master 127.19.34.126:45351 requested a full tablet report, sending...
I20260619 14:54:08.496562 885 ts_manager.cc:194] Registered new tserver with Master: bd5de741a6004d7cb5132cbe2e86365d (127.19.34.65:44209)
I20260619 14:54:08.498262 885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:51299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
I20260619 14:54:08.506605 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:45351
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:45559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:08.603367 1083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:08.603606 1083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:08.603690 1083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:08.607369 1083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:08.607416 1083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:08.607520 1083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:54:08.611855 1083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:45559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:45351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:08.612706 1083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:08.613560 1083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:08.621971 1089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:08.621977 1088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:08.621977 1083 server_base.cc:1034] running on GCE node
W20260619 14:54:08.622143 1091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:08.622609 1083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:08.623196 1083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:08.624351 1083 hybrid_clock.cc:648] HybridClock initialized: now 1781880848624337 us; error 22 us; skew 500 ppm
Jun 19 14:54:08 dist-test-slave-3qdh krb5kdc[842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880848, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:08.627082 1083 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:54:08.627887 1083 webserver.cc:460] Webserver started at http://127.19.34.66:37287/ using document root <none> and password file <none>
I20260619 14:54:08.628430 1083 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:08.628510 1083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:08.628707 1083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:08.630511 1083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9"
format_stamp: "Formatted at 2026-06-19 14:54:08 on dist-test-slave-3qdh"
server_key: "04959784ced27dadcc740abc2c93b07a"
server_key_iv: "b563053cd03271b1a36da10f13a9a659"
server_key_version: "encryptionkey@0"
I20260619 14:54:08.631016 1083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9"
format_stamp: "Formatted at 2026-06-19 14:54:08 on dist-test-slave-3qdh"
server_key: "04959784ced27dadcc740abc2c93b07a"
server_key_iv: "b563053cd03271b1a36da10f13a9a659"
server_key_version: "encryptionkey@0"
I20260619 14:54:08.634388 1083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:54:08.636400 1098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:08.637465 1083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:54:08.637585 1083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9"
format_stamp: "Formatted at 2026-06-19 14:54:08 on dist-test-slave-3qdh"
server_key: "04959784ced27dadcc740abc2c93b07a"
server_key_iv: "b563053cd03271b1a36da10f13a9a659"
server_key_version: "encryptionkey@0"
I20260619 14:54:08.637678 1083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:08.649785 1083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:08.652227 1083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:08.652412 1083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:08.652910 1083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:08.653647 1083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:08.653708 1083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:08.653785 1083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:08.653829 1083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:08.661522 1083 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:39079
I20260619 14:54:08.661545 1211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:39079 every 8 connection(s)
I20260619 14:54:08.662231 1083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
Jun 19 14:54:08 dist-test-slave-3qdh krb5kdc[842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880848, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:54:08.671355 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 1083
I20260619 14:54:08.671496 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:54:08.671746 19593 external_mini_cluster.cc:1428] Setting key 2ebfbdaee4f85787e65e209606b99a50
I20260619 14:54:08.673951 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:08.663740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:46829 (local address 127.19.34.126:45351)
0619 14:54:08.663898 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:08.663901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:08.664627 (+ 726us) server_negotiation.cc:407] Connection header received
0619 14:54:08.665867 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:08.665869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:08.665913 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:08.665984 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:08.666964 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.667399 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.668370 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.668570 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.668648 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:08.670569 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:08.670572 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:08.670573 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:08.670598 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:08.671980 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:08.672518 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:08.672540 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:08.672541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:08.672621 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:08.672939 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:08.672942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:08.672944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:08.673125 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:08.673210 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:08.673618 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 14:54:08.673745 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:54:08.674391 1214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:08.664089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.66:46829)
0619 14:54:08.664489 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:08.664525 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:08.665689 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:08.666126 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:08.666132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:08.666456 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:08.666862 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:08.666872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.667482 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.667484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:08.668262 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:08.668269 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.668683 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.668685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:08.668801 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:08.669235 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:08.669249 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:08.670442 (+ 1193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:08.672074 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:08.672079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:08.672095 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:08.672410 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:08.672708 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:08.672711 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:08.672713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:08.672868 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:08.673229 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:08.673233 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:08.673407 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 14:54:08.673528 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 14:54:08.673718 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":102,"threads_started":1}
I20260619 14:54:08.675343 1212 heartbeater.cc:347] Connected to a master server at 127.19.34.126:45351
I20260619 14:54:08.675576 1212 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:08.675956 1212 heartbeater.cc:511] Master 127.19.34.126:45351 requested a full tablet report, sending...
I20260619 14:54:08.676860 885 ts_manager.cc:194] Registered new tserver with Master: 530f0dc8c0ee46bcac6b01f0ee984dd9 (127.19.34.66:39079)
I20260619 14:54:08.677336 885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:46829
WARNING: no policy specified for kudu/127.19.34.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
I20260619 14:54:08.696393 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:45351
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:45559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:08.794350 1217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:08.794543 1217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:08.794576 1217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:08.798084 1217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:08.798126 1217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:08.798172 1217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:54:08.802459 1217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:45559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:45351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:08.803198 1217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:08.803990 1217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:08.812517 1222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:08.812672 1223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:08.812636 1225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:08.814297 1217 server_base.cc:1034] running on GCE node
I20260619 14:54:08.814667 1217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:08.815165 1217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:08.816322 1217 hybrid_clock.cc:648] HybridClock initialized: now 1781880848816277 us; error 52 us; skew 500 ppm
Jun 19 14:54:08 dist-test-slave-3qdh krb5kdc[842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880848, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:08.818802 1217 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:54:08.819514 1217 webserver.cc:460] Webserver started at http://127.19.34.67:44501/ using document root <none> and password file <none>
I20260619 14:54:08.819957 1217 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:08.820008 1217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:08.820197 1217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:08.821897 1217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "e1d41dd2c3174f229235816f3c80030c"
format_stamp: "Formatted at 2026-06-19 14:54:08 on dist-test-slave-3qdh"
server_key: "ba95c513caaed9ed0a0c1ae3d3aebfae"
server_key_iv: "0084569e94c1a47488df92d67090fdd5"
server_key_version: "encryptionkey@0"
I20260619 14:54:08.822386 1217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "e1d41dd2c3174f229235816f3c80030c"
format_stamp: "Formatted at 2026-06-19 14:54:08 on dist-test-slave-3qdh"
server_key: "ba95c513caaed9ed0a0c1ae3d3aebfae"
server_key_iv: "0084569e94c1a47488df92d67090fdd5"
server_key_version: "encryptionkey@0"
I20260619 14:54:08.825641 1217 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:54:08.827700 1232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:08.828785 1217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:54:08.828912 1217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "e1d41dd2c3174f229235816f3c80030c"
format_stamp: "Formatted at 2026-06-19 14:54:08 on dist-test-slave-3qdh"
server_key: "ba95c513caaed9ed0a0c1ae3d3aebfae"
server_key_iv: "0084569e94c1a47488df92d67090fdd5"
server_key_version: "encryptionkey@0"
I20260619 14:54:08.829011 1217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:08.840209 1217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:08.842696 1217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:08.842864 1217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:08.843307 1217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:08.843940 1217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:08.843978 1217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:08.844030 1217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:08.844060 1217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:08.852126 1217 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:44171
I20260619 14:54:08.852136 1345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:44171 every 8 connection(s)
I20260619 14:54:08.852862 1217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
Jun 19 14:54:08 dist-test-slave-3qdh krb5kdc[842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880848, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:54:08.861119 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 1217
I20260619 14:54:08.861217 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:54:08.861443 19593 external_mini_cluster.cc:1428] Setting key 90bfef39e084f3c7202630c9f9849584
I20260619 14:54:08.864059 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:08.854359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:51215 (local address 127.19.34.126:45351)
0619 14:54:08.854480 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:08.854483 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:08.855133 (+ 650us) server_negotiation.cc:407] Connection header received
0619 14:54:08.856401 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:08.856403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:08.856449 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:08.856531 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:08.857438 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.857916 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.858797 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.858974 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.859039 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:08.860896 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:08.860899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:08.860901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:08.860923 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:08.862346 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:08.862832 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:08.862835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:08.862837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:08.862922 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:08.863267 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:08.863269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:08.863270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:08.863412 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:08.863479 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:08.863884 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 14:54:08.863964 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:54:08.864725 1348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:08.854639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.67:51215)
0619 14:54:08.855003 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:08.855037 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:08.856177 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:08.856679 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:08.856685 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:08.856961 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:08.857330 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:08.857340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.858000 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.858001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:08.858709 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:08.858716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.859072 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.859074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:08.859183 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:08.859559 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:08.859572 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:08.860764 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:08.862456 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:08.862460 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:08.862468 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:08.862742 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:08.863001 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:08.863003 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:08.863004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:08.863170 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:08.863486 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:08.863489 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:08.863674 (+ 185us) client_negotiation.cc:769] Sending connection context
0619 14:54:08.863848 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 14:54:08.864036 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":93,"threads_started":1}
I20260619 14:54:08.865643 1346 heartbeater.cc:347] Connected to a master server at 127.19.34.126:45351
I20260619 14:54:08.865839 1346 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:08.866235 1346 heartbeater.cc:511] Master 127.19.34.126:45351 requested a full tablet report, sending...
I20260619 14:54:08.867058 885 ts_manager.cc:194] Registered new tserver with Master: e1d41dd2c3174f229235816f3c80030c (127.19.34.67:44171)
I20260619 14:54:08.867513 885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:51215
I20260619 14:54:08.874352 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:54:08.886431 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:08.875993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45752 (local address 127.19.34.126:45351)
0619 14:54:08.876096 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:08.876099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:08.876278 (+ 179us) server_negotiation.cc:407] Connection header received
0619 14:54:08.876402 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:08.876405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:08.876438 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:08.876502 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:08.877037 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.877440 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.878089 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.878251 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.878313 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:08.884133 (+ 5820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:08.884135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:08.884137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:08.884154 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:08.885221 (+ 1067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:08.885566 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:08.885568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:08.885569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:08.885634 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:08.885835 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:08.885837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:08.885838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:08.886002 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:08.886114 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:08.886263 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 14:54:08.886338 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:54:08.888677 885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:54:08.891089 885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:54:08.903368 1359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:08.898357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:44209 (local address 127.0.0.1:42502)
0619 14:54:08.898676 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:08.898688 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:08.898767 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:08.899363 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:08.899366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:08.899382 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:08.899518 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:08.899526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.900665 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.900670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:08.902099 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:08.902105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.902804 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.902807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:08.902969 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:08.903012 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 14:54:08.903092 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 14:54:08.903237 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":265,"thread_start_us":70,"threads_started":1}
I20260619 14:54:08.903373 1361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:08.898616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42502 (local address 127.19.34.65:44209)
0619 14:54:08.898965 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:08.898976 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:08.898997 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:54:08.899063 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:08.899067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:08.899232 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:08.899655 (+ 423us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:08.899772 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.900556 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.902221 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.902683 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.902942 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:08.903071 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:08.903126 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 14:54:08.903204 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":127,"threads_started":1}
I20260619 14:54:08.903498 1362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:08.898934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58684 (local address 127.19.34.67:44171)
0619 14:54:08.899247 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:08.899257 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:08.899530 (+ 273us) server_negotiation.cc:407] Connection header received
0619 14:54:08.899590 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:08.899594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:08.899738 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:08.899868 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:08.900062 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.900928 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.902001 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.902668 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.902762 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:08.902868 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:08.903307 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 14:54:08.903379 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":86,"threads_started":1}
I20260619 14:54:08.903700 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:08.898204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60608 (local address 127.19.34.66:39079)
0619 14:54:08.898734 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:08.898744 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:08.898764 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:54:08.898948 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:08.898954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:08.899138 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:08.899504 (+ 366us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:08.899658 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.900682 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.902203 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.902852 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.903307 (+ 455us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:08.903418 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:08.903464 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 14:54:08.903558 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":89,"threads_started":1}
I20260619 14:54:08.904047 1360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:08.898649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:44171 (local address 127.0.0.1:58684)
0619 14:54:08.899353 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:08.899364 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:08.899454 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:08.899865 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:08.899867 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:08.899882 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:08.899951 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:08.899956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.901026 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.901029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:08.901877 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:08.901887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.902782 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.902788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:08.902950 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:08.903012 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 14:54:08.903101 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 14:54:08.903197 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":75,"thread_start_us":70,"threads_started":1}
I20260619 14:54:08.904069 1357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:08.898074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:39079 (local address 127.0.0.1:60608)
0619 14:54:08.898338 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:08.898373 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:08.898466 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:08.899291 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:08.899296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:08.899317 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:08.899482 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:08.899492 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.900790 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.900793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:08.902096 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:08.902105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:08.902965 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:08.902968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:08.903040 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:08.903055 (+ 15us) client_negotiation.cc:769] Sending connection context
0619 14:54:08.903093 (+ 38us) client_negotiation.cc:240] Negotiation successful
0619 14:54:08.903237 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":100,"thread_start_us":80,"threads_started":1}
I20260619 14:54:08.906322 1012 tablet_service.cc:1467] Processing CreateTablet for tablet 5434e8de9a004eb289a03d8d1484a882 (DEFAULT_TABLE table=test-table [id=48b6a421b1974172838d38c3b92cc949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:08.906504 1280 tablet_service.cc:1467] Processing CreateTablet for tablet 5434e8de9a004eb289a03d8d1484a882 (DEFAULT_TABLE table=test-table [id=48b6a421b1974172838d38c3b92cc949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:08.906503 1146 tablet_service.cc:1467] Processing CreateTablet for tablet 5434e8de9a004eb289a03d8d1484a882 (DEFAULT_TABLE table=test-table [id=48b6a421b1974172838d38c3b92cc949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:08.907207 1012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5434e8de9a004eb289a03d8d1484a882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:08.907384 1146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5434e8de9a004eb289a03d8d1484a882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:08.907388 1280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5434e8de9a004eb289a03d8d1484a882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:08.912782 1363 tablet_bootstrap.cc:492] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d: Bootstrap starting.
I20260619 14:54:08.913326 1364 tablet_bootstrap.cc:492] T 5434e8de9a004eb289a03d8d1484a882 P e1d41dd2c3174f229235816f3c80030c: Bootstrap starting.
I20260619 14:54:08.914268 1365 tablet_bootstrap.cc:492] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9: Bootstrap starting.
I20260619 14:54:08.915355 1364 tablet_bootstrap.cc:654] T 5434e8de9a004eb289a03d8d1484a882 P e1d41dd2c3174f229235816f3c80030c: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:08.915359 1363 tablet_bootstrap.cc:654] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:08.916281 1363 log.cc:824] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:08.916407 1365 tablet_bootstrap.cc:654] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:08.916566 1364 log.cc:824] T 5434e8de9a004eb289a03d8d1484a882 P e1d41dd2c3174f229235816f3c80030c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:08.917268 1365 log.cc:824] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:08.918198 1363 tablet_bootstrap.cc:492] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d: No bootstrap required, opened a new log
I20260619 14:54:08.918282 1364 tablet_bootstrap.cc:492] T 5434e8de9a004eb289a03d8d1484a882 P e1d41dd2c3174f229235816f3c80030c: No bootstrap required, opened a new log
I20260619 14:54:08.918365 1363 ts_tablet_manager.cc:1397] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 14:54:08.918457 1364 ts_tablet_manager.cc:1397] T 5434e8de9a004eb289a03d8d1484a882 P e1d41dd2c3174f229235816f3c80030c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:54:08.918897 1365 tablet_bootstrap.cc:492] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9: No bootstrap required, opened a new log
I20260619 14:54:08.919044 1365 ts_tablet_manager.cc:1397] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:54:08.921023 1364 raft_consensus.cc:348] T 5434e8de9a004eb289a03d8d1484a882 P e1d41dd2c3174f229235816f3c80030c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:08.921216 1364 raft_consensus.cc:374] T 5434e8de9a004eb289a03d8d1484a882 P e1d41dd2c3174f229235816f3c80030c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:08.921276 1364 raft_consensus.cc:729] T 5434e8de9a004eb289a03d8d1484a882 P e1d41dd2c3174f229235816f3c80030c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1d41dd2c3174f229235816f3c80030c, State: Initialized, Role: FOLLOWER
I20260619 14:54:08.921641 1364 consensus_queue.cc:260] T 5434e8de9a004eb289a03d8d1484a882 P e1d41dd2c3174f229235816f3c80030c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:08.921701 1363 raft_consensus.cc:348] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:08.921938 1363 raft_consensus.cc:374] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:08.921999 1363 raft_consensus.cc:729] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd5de741a6004d7cb5132cbe2e86365d, State: Initialized, Role: FOLLOWER
I20260619 14:54:08.922168 1365 raft_consensus.cc:348] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:08.922399 1365 raft_consensus.cc:374] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:08.922354 1363 consensus_queue.cc:260] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:08.922513 1365 raft_consensus.cc:729] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530f0dc8c0ee46bcac6b01f0ee984dd9, State: Initialized, Role: FOLLOWER
I20260619 14:54:08.922523 1364 ts_tablet_manager.cc:1428] T 5434e8de9a004eb289a03d8d1484a882 P e1d41dd2c3174f229235816f3c80030c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 14:54:08.922809 1346 heartbeater.cc:503] Master 127.19.34.126:45351 was elected leader, sending a full tablet report...
I20260619 14:54:08.922948 1365 consensus_queue.cc:260] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:08.923446 1212 heartbeater.cc:503] Master 127.19.34.126:45351 was elected leader, sending a full tablet report...
I20260619 14:54:08.923720 1365 ts_tablet_manager.cc:1428] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:54:08.924032 1363 ts_tablet_manager.cc:1428] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260619 14:54:08.924163 1078 heartbeater.cc:503] Master 127.19.34.126:45351 was elected leader, sending a full tablet report...
W20260619 14:54:08.981307 1079 tablet.cc:2369] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:54:09.006351 1371 raft_consensus.cc:482] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:54:09.006558 1371 raft_consensus.cc:504] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:09.007620 1371 leader_election.cc:288] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1d41dd2c3174f229235816f3c80030c (127.19.34.67:44171), bd5de741a6004d7cb5132cbe2e86365d (127.19.34.65:44209)
I20260619 14:54:09.010849 1214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.007832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:44171 (local address 127.19.34.66:47195)
0619 14:54:09.007950 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:09.007964 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:09.008050 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:09.008291 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:09.008311 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:09.008328 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:09.008427 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:09.008432 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.009251 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.009257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:09.009922 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:09.009929 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.010470 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.010473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:09.010568 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:09.010585 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 14:54:09.010671 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 14:54:09.010736 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260619 14:54:09.010849 1362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.007950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:47195 (local address 127.19.34.67:44171)
0619 14:54:09.008075 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:09.008084 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:09.008096 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:54:09.008157 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:09.008159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:09.008193 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:09.008250 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:09.008533 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.009163 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.010034 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.010371 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.010479 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:09.010527 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:09.010689 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 14:54:09.010746 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:54:09.011157 1361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.007935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:57359 (local address 127.19.34.65:44209)
0619 14:54:09.008041 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:09.008046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:09.008555 (+ 509us) server_negotiation.cc:407] Connection header received
0619 14:54:09.008622 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:09.008625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:09.008681 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:09.008748 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:09.009001 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.009448 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.010308 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.010661 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.010770 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:09.010813 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:09.010991 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 14:54:09.011058 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260619 14:54:09.011442 1372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.007857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:44209 (local address 127.19.34.66:57359)
0619 14:54:09.008445 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:09.008454 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:09.008553 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:09.008769 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:09.008771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:09.008786 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:09.008894 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:09.008899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.009556 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.009559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:09.010196 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:09.010203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.010763 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.010766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:09.010849 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:09.010871 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:54:09.011289 (+ 418us) client_negotiation.cc:240] Negotiation successful
0619 14:54:09.011353 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":245,"thread_start_us":115,"threads_started":1}
I20260619 14:54:09.011418 1300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5434e8de9a004eb289a03d8d1484a882" candidate_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1d41dd2c3174f229235816f3c80030c" is_pre_election: true
I20260619 14:54:09.011651 1300 raft_consensus.cc:2454] T 5434e8de9a004eb289a03d8d1484a882 P e1d41dd2c3174f229235816f3c80030c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530f0dc8c0ee46bcac6b01f0ee984dd9 in term 0.
I20260619 14:54:09.012006 1032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5434e8de9a004eb289a03d8d1484a882" candidate_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd5de741a6004d7cb5132cbe2e86365d" is_pre_election: true
I20260619 14:54:09.012048 1101 leader_election.cc:302] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530f0dc8c0ee46bcac6b01f0ee984dd9, e1d41dd2c3174f229235816f3c80030c; no voters:
I20260619 14:54:09.012270 1032 raft_consensus.cc:2454] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 530f0dc8c0ee46bcac6b01f0ee984dd9 in term 0.
I20260619 14:54:09.012285 1371 raft_consensus.cc:2781] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:54:09.012375 1371 raft_consensus.cc:482] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:54:09.012411 1371 raft_consensus.cc:3037] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:09.013406 1371 raft_consensus.cc:504] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:09.013700 1371 leader_election.cc:288] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [CANDIDATE]: Term 1 election: Requested vote from peers e1d41dd2c3174f229235816f3c80030c (127.19.34.67:44171), bd5de741a6004d7cb5132cbe2e86365d (127.19.34.65:44209)
I20260619 14:54:09.013970 1032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5434e8de9a004eb289a03d8d1484a882" candidate_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd5de741a6004d7cb5132cbe2e86365d"
I20260619 14:54:09.013965 1300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5434e8de9a004eb289a03d8d1484a882" candidate_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1d41dd2c3174f229235816f3c80030c"
I20260619 14:54:09.014074 1300 raft_consensus.cc:3037] T 5434e8de9a004eb289a03d8d1484a882 P e1d41dd2c3174f229235816f3c80030c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:09.014096 1032 raft_consensus.cc:3037] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:09.015234 1032 raft_consensus.cc:2454] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530f0dc8c0ee46bcac6b01f0ee984dd9 in term 1.
I20260619 14:54:09.015234 1300 raft_consensus.cc:2454] T 5434e8de9a004eb289a03d8d1484a882 P e1d41dd2c3174f229235816f3c80030c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 530f0dc8c0ee46bcac6b01f0ee984dd9 in term 1.
I20260619 14:54:09.015513 1099 leader_election.cc:302] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530f0dc8c0ee46bcac6b01f0ee984dd9, bd5de741a6004d7cb5132cbe2e86365d; no voters:
I20260619 14:54:09.015717 1371 raft_consensus.cc:2781] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:09.015925 1371 raft_consensus.cc:686] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 1 LEADER]: Becoming Leader. State: Replica: 530f0dc8c0ee46bcac6b01f0ee984dd9, State: Running, Role: LEADER
I20260619 14:54:09.016239 1371 consensus_queue.cc:237] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:09.019258 886 catalog_manager.cc:5495] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 530f0dc8c0ee46bcac6b01f0ee984dd9 (127.19.34.66). New cstate: current_term: 1 leader_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } health_report { overall_health: UNKNOWN } } }
I20260619 14:54:09.055083 1358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.051821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60610 (local address 127.19.34.66:39079)
0619 14:54:09.051937 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:09.051942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:09.051985 (+ 43us) server_negotiation.cc:407] Connection header received
0619 14:54:09.052151 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:09.052154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:09.052202 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:09.052295 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:54:09.052536 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.053013 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.053784 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.053977 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.054070 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:09.054329 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:54:09.054671 (+ 342us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:54:09.054814 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:09.054879 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 14:54:09.054928 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:54:09.060292 1300 raft_consensus.cc:1261] T 5434e8de9a004eb289a03d8d1484a882 P e1d41dd2c3174f229235816f3c80030c [term 1 FOLLOWER]: Refusing update from remote peer 530f0dc8c0ee46bcac6b01f0ee984dd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:09.060287 1032 raft_consensus.cc:1261] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Refusing update from remote peer 530f0dc8c0ee46bcac6b01f0ee984dd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:09.060969 1371 consensus_queue.cc:1035] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:09.061160 1373 consensus_queue.cc:1035] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:09.067327 1376 mvcc.cc:205] Tried to move back new op lower bound from 7298583957744078848 to 7298583957572968448. Current Snapshot: MvccSnapshot[applied={T|T < 7298583957744078848}]
I20260619 14:54:09.067538 1378 mvcc.cc:205] Tried to move back new op lower bound from 7298583957744078848 to 7298583957572968448. Current Snapshot: MvccSnapshot[applied={T|T < 7298583957744078848}]
I20260619 14:54:09.073990 887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:09.074101 887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:09.076207 887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5: Sending DeleteTablet for 3 replicas of tablet 5434e8de9a004eb289a03d8d1484a882
I20260619 14:54:09.076891 1280 tablet_service.cc:1514] Processing DeleteTablet for tablet 5434e8de9a004eb289a03d8d1484a882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:09 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:58684
I20260619 14:54:09.076889 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet 5434e8de9a004eb289a03d8d1484a882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:09 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:42502
I20260619 14:54:09.077057 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 949
I20260619 14:54:09.077268 1146 tablet_service.cc:1514] Processing DeleteTablet for tablet 5434e8de9a004eb289a03d8d1484a882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:09 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:60608
I20260619 14:54:09.077718 1385 tablet_replica.cc:330] T 5434e8de9a004eb289a03d8d1484a882 P e1d41dd2c3174f229235816f3c80030c: stopping tablet replica
I20260619 14:54:09.077982 1385 raft_consensus.cc:2229] T 5434e8de9a004eb289a03d8d1484a882 P e1d41dd2c3174f229235816f3c80030c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:54:09.078055 1386 tablet_replica.cc:330] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9: stopping tablet replica
I20260619 14:54:09.078298 1386 raft_consensus.cc:2229] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:54:09.078330 1385 raft_consensus.cc:2258] T 5434e8de9a004eb289a03d8d1484a882 P e1d41dd2c3174f229235816f3c80030c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:09.078577 1386 raft_consensus.cc:2258] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:09.079754 1386 ts_tablet_manager.cc:1905] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:09.079754 1385 ts_tablet_manager.cc:1905] T 5434e8de9a004eb289a03d8d1484a882 P e1d41dd2c3174f229235816f3c80030c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:09.082130 1385 ts_tablet_manager.cc:1918] T 5434e8de9a004eb289a03d8d1484a882 P e1d41dd2c3174f229235816f3c80030c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:09.082150 1386 ts_tablet_manager.cc:1918] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:09.082221 1386 log.cc:1196] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/wal/wals/5434e8de9a004eb289a03d8d1484a882
I20260619 14:54:09.082274 1385 log.cc:1196] T 5434e8de9a004eb289a03d8d1484a882 P e1d41dd2c3174f229235816f3c80030c: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/5434e8de9a004eb289a03d8d1484a882
I20260619 14:54:09.082561 1386 ts_tablet_manager.cc:1939] T 5434e8de9a004eb289a03d8d1484a882 P 530f0dc8c0ee46bcac6b01f0ee984dd9: Deleting consensus metadata
I20260619 14:54:09.082579 1385 ts_tablet_manager.cc:1939] T 5434e8de9a004eb289a03d8d1484a882 P e1d41dd2c3174f229235816f3c80030c: Deleting consensus metadata
I20260619 14:54:09.083487 873 catalog_manager.cc:4841] TS 530f0dc8c0ee46bcac6b01f0ee984dd9 (127.19.34.66:39079): tablet 5434e8de9a004eb289a03d8d1484a882 (table test-table [id=48b6a421b1974172838d38c3b92cc949]) successfully deleted
I20260619 14:54:09.083648 873 catalog_manager.cc:4841] TS e1d41dd2c3174f229235816f3c80030c (127.19.34.67:44171): tablet 5434e8de9a004eb289a03d8d1484a882 (table test-table [id=48b6a421b1974172838d38c3b92cc949]) successfully deleted
W20260619 14:54:09.084481 871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.34.65:44209 (error 108)
I20260619 14:54:09.085120 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 1083
I20260619 14:54:09.085265 1360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.085017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:44209 (local address 127.0.0.1:42504)
0619 14:54:09.085112 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:09.085188 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.65:44209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 14:54:09.086927 871 catalog_manager.cc:4584] TS bd5de741a6004d7cb5132cbe2e86365d (127.19.34.65:44209): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5434e8de9a004eb289a03d8d1484a882: Network error: Client connection negotiation failed: client connection to 127.19.34.65:44209: connect: Connection refused (error 111)
I20260619 14:54:09.089264 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 1217
I20260619 14:54:09.094162 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 855
I20260619 14:54:09.099720 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:45351
--webserver_interface=127.19.34.126
--webserver_port=38623
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:45559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:45351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:09.193233 1388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:09.193426 1388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:09.193459 1388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:09.197117 1388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:09.197157 1388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:09.197173 1388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:09.197224 1388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:09.197248 1388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:09.201627 1388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:45559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:45351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:45351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=38623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:09.202455 1388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:09.203246 1388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:09.214133 1395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:09.214161 1397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:09.214270 1388 server_base.cc:1034] running on GCE node
W20260619 14:54:09.214169 1394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:09.214792 1388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:09.215770 1388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:09.216938 1388 hybrid_clock.cc:648] HybridClock initialized: now 1781880849216920 us; error 28 us; skew 500 ppm
Jun 19 14:54:09 dist-test-slave-3qdh krb5kdc[842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880849, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:09.219370 1388 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:54:09.220098 1388 webserver.cc:460] Webserver started at http://127.19.34.126:38623/ using document root <none> and password file <none>
I20260619 14:54:09.220633 1388 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:09.220726 1388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:09.223891 1388 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:54:09.225797 1404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:09.226902 1388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:54:09.227020 1388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "5fda20c76804435e88c1573a73fc0ab5"
format_stamp: "Formatted at 2026-06-19 14:54:08 on dist-test-slave-3qdh"
server_key: "da42ef64eca699ce16385e16099e1951"
server_key_iv: "fe4227969ffa66f60a48443485ab6221"
server_key_version: "encryptionkey@0"
I20260619 14:54:09.227516 1388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:09.239472 1388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:09.242138 1388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:09.242311 1388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:09.248728 1388 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:45351
I20260619 14:54:09.249003 1456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:45351 every 8 connection(s)
I20260619 14:54:09.249634 1388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:09.253091 1457 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:54:09.254508 1457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5: Bootstrap starting.
I20260619 14:54:09.254783 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 1388
I20260619 14:54:09.255148 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:44209
--local_ip_for_outbound_sockets=127.19.34.65
--tserver_master_addrs=127.19.34.126:45351
--webserver_port=34327
--webserver_interface=127.19.34.65
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:45559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:54:09.274185 1457 log.cc:824] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:09.281807 1457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:54:09.282111 1457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5: Bootstrap complete.
I20260619 14:54:09.285652 1457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fda20c76804435e88c1573a73fc0ab5" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45351 } }
I20260619 14:54:09.286067 1457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fda20c76804435e88c1573a73fc0ab5, State: Initialized, Role: FOLLOWER
I20260619 14:54:09.286497 1457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fda20c76804435e88c1573a73fc0ab5" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45351 } }
I20260619 14:54:09.286649 1457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:09.286734 1457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:09.286829 1457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:54:09.288431 1457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fda20c76804435e88c1573a73fc0ab5" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45351 } }
I20260619 14:54:09.288712 1457 leader_election.cc:302] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fda20c76804435e88c1573a73fc0ab5; no voters:
I20260619 14:54:09.288981 1457 leader_election.cc:288] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:54:09.289045 1461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:54:09.289294 1461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 2 LEADER]: Becoming Leader. State: Replica: 5fda20c76804435e88c1573a73fc0ab5, State: Running, Role: LEADER
I20260619 14:54:09.289688 1461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fda20c76804435e88c1573a73fc0ab5" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45351 } }
I20260619 14:54:09.290117 1457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:09.290460 1461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fda20c76804435e88c1573a73fc0ab5. Latest consensus state: current_term: 2 leader_uuid: "5fda20c76804435e88c1573a73fc0ab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fda20c76804435e88c1573a73fc0ab5" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45351 } } }
I20260619 14:54:09.291113 1461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:09.291002 1462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5fda20c76804435e88c1573a73fc0ab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fda20c76804435e88c1573a73fc0ab5" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45351 } } }
I20260619 14:54:09.291630 1462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:09.291862 1470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:09.296134 1470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:09.296746 1470 catalog_manager.cc:1259] Loaded cluster ID: 3683abfa882f484c8883630a35127119
I20260619 14:54:09.296799 1470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:09.297992 1470 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 14:54:09.302373 1470 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 14:54:09.307126 1470 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:09.307740 1470 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5: Loaded TSK: 0
I20260619 14:54:09.308903 1470 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:54:09.363830 1459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:09.364034 1459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:09.364071 1459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:09.367748 1459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:09.367794 1459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:09.367892 1459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:54:09.372458 1459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:45559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:44209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=34327
--tserver_master_addrs=127.19.34.126:45351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:09.373317 1459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:09.374182 1459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:09.382745 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:09.382769 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:09.382741 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:09.382970 1459 server_base.cc:1034] running on GCE node
I20260619 14:54:09.383400 1459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:09.384008 1459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:09.385172 1459 hybrid_clock.cc:648] HybridClock initialized: now 1781880849385155 us; error 32 us; skew 500 ppm
Jun 19 14:54:09 dist-test-slave-3qdh krb5kdc[842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880849, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:09.387622 1459 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:54:09.388535 1459 webserver.cc:460] Webserver started at http://127.19.34.65:34327/ using document root <none> and password file <none>
I20260619 14:54:09.389019 1459 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:09.389101 1459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:09.392562 1459 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 14:54:09.394433 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:09.395569 1459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:54:09.395690 1459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "bd5de741a6004d7cb5132cbe2e86365d"
format_stamp: "Formatted at 2026-06-19 14:54:08 on dist-test-slave-3qdh"
server_key: "bb42d1c75047b6999106dd06f2602935"
server_key_iv: "3f28884ec4939ad766f43d8358741d5c"
server_key_version: "encryptionkey@0"
I20260619 14:54:09.396183 1459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:09.419836 1459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:09.422435 1459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:09.422595 1459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:09.423134 1459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:09.424088 1501 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:54:09.427805 1459 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:54:09.427875 1459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 14:54:09.427958 1459 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:54:09.429463 1459 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:54:09.429520 1459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260619 14:54:09.429774 1501 tablet_bootstrap.cc:492] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d: Bootstrap starting.
I20260619 14:54:09.439160 1459 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:44209
I20260619 14:54:09.439229 1608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:44209 every 8 connection(s)
I20260619 14:54:09.439960 1459 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:54:09.441322 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 1459
I20260619 14:54:09.441684 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:39079
--local_ip_for_outbound_sockets=127.19.34.66
--tserver_master_addrs=127.19.34.126:45351
--webserver_port=37287
--webserver_interface=127.19.34.66
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:45559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:54:09.447182 1501 log.cc:824] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:54:09 dist-test-slave-3qdh krb5kdc[842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880849, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:54:09.454658 1611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.441722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.65:36353)
0619 14:54:09.442299 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:09.442349 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:09.443358 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:09.444399 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:09.444406 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:09.444781 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:09.445166 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:09.445178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.446459 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.446463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:09.447233 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:09.447241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.448170 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.448176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:09.448377 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:09.448962 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:09.448989 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:09.450239 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:09.451987 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:09.451991 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:09.452002 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:09.452256 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:09.452623 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:09.452626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:09.452627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:09.452746 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:09.453243 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:09.453248 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:09.453453 (+ 205us) client_negotiation.cc:769] Sending connection context
0619 14:54:09.453603 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 14:54:09.453893 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":196,"threads_started":1}
I20260619 14:54:09.454648 1613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.442938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:36353 (local address 127.19.34.126:45351)
0619 14:54:09.443352 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:09.443369 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:09.443412 (+ 43us) server_negotiation.cc:407] Connection header received
0619 14:54:09.443766 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:09.443776 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:09.444043 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:09.444353 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:09.445320 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.446320 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.447365 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.448010 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.448218 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:09.450400 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:09.450406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:09.450423 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:09.450446 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:09.451871 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:09.452420 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:09.452425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:09.452433 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:09.452540 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:09.452850 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:09.452853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:09.452854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:09.453136 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:09.453267 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:09.453715 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 14:54:09.453938 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":114,"threads_started":1}
I20260619 14:54:09.455508 1501 tablet_bootstrap.cc:492] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:54:09.455901 1501 tablet_bootstrap.cc:492] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d: Bootstrap complete.
I20260619 14:54:09.457046 1609 heartbeater.cc:347] Connected to a master server at 127.19.34.126:45351
I20260619 14:54:09.457226 1501 ts_tablet_manager.cc:1397] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.011s
I20260619 14:54:09.457237 1609 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:09.457664 1609 heartbeater.cc:511] Master 127.19.34.126:45351 requested a full tablet report, sending...
I20260619 14:54:09.459674 1421 ts_manager.cc:194] Registered new tserver with Master: bd5de741a6004d7cb5132cbe2e86365d (127.19.34.65:44209)
I20260619 14:54:09.459878 1501 raft_consensus.cc:348] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:09.460330 1501 raft_consensus.cc:729] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd5de741a6004d7cb5132cbe2e86365d, State: Initialized, Role: FOLLOWER
I20260619 14:54:09.460847 1501 consensus_queue.cc:260] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:09.461695 1501 ts_tablet_manager.cc:1428] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 14:54:09.462860 1421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:36353
I20260619 14:54:09.464986 1609 heartbeater.cc:503] Master 127.19.34.126:45351 was elected leader, sending a full tablet report...
Jun 19 14:54:09 dist-test-slave-3qdh krb5kdc[842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880849, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for kudu/127.19.34.65@KRBTEST.COM
I20260619 14:54:09.472100 1617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.462828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42512 (local address 127.19.34.65:44209)
0619 14:54:09.463079 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:09.463089 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:09.463106 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:54:09.463158 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:09.463162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:09.463319 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:09.463433 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:09.464036 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.465216 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.466427 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.466798 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.466869 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:09.469032 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:09.469039 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:09.469047 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:09.469065 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:09.470456 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:09.470905 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:09.470909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:09.470910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:09.470973 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:09.471246 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:09.471248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:09.471249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:09.471451 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:09.471540 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:09.471875 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 14:54:09.471987 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":174,"thread_start_us":93,"threads_started":1}
I20260619 14:54:09.472124 1616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.462251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:44209 (local address 127.0.0.1:42512)
0619 14:54:09.462575 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:09.462618 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:09.462739 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:09.463454 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:09.463460 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:09.463802 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:09.463905 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:09.463912 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.465361 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.465367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:09.466280 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:09.466306 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.466898 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.466902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:09.466997 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:09.467419 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:09.467432 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:09.468920 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:09.470550 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:09.470559 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:09.470563 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:09.470816 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:09.471046 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:09.471049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:09.471050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:09.471159 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:09.471547 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:09.471552 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:09.471672 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 14:54:09.471804 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 14:54:09.471969 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":79,"threads_started":1}
I20260619 14:54:09.472968 1543 tablet_service.cc:1514] Processing DeleteTablet for tablet 5434e8de9a004eb289a03d8d1484a882 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:09 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:42512
I20260619 14:54:09.473614 1618 tablet_replica.cc:330] stopping tablet replica
I20260619 14:54:09.473855 1618 raft_consensus.cc:2229] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:54:09.474059 1618 raft_consensus.cc:2258] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:09.474929 1618 ts_tablet_manager.cc:1905] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:09.478255 1618 ts_tablet_manager.cc:1918] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:09.478324 1618 log.cc:1196] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/wal/wals/5434e8de9a004eb289a03d8d1484a882
I20260619 14:54:09.478615 1618 ts_tablet_manager.cc:1939] T 5434e8de9a004eb289a03d8d1484a882 P bd5de741a6004d7cb5132cbe2e86365d: Deleting consensus metadata
I20260619 14:54:09.479918 1405 catalog_manager.cc:4841] TS bd5de741a6004d7cb5132cbe2e86365d (127.19.34.65:44209): tablet 5434e8de9a004eb289a03d8d1484a882 (table test-table [id=48b6a421b1974172838d38c3b92cc949]) successfully deleted
W20260619 14:54:09.549734 1612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:09.550067 1612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:09.550122 1612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:09.554721 1612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:09.554769 1612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:09.554821 1612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:54:09.559531 1612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:45559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:39079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=37287
--tserver_master_addrs=127.19.34.126:45351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:09.560251 1612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:09.561044 1612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:09.568974 1623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:09.569000 1624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:09.568975 1626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:09.569195 1612 server_base.cc:1034] running on GCE node
I20260619 14:54:09.569578 1612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:09.570088 1612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:09.571245 1612 hybrid_clock.cc:648] HybridClock initialized: now 1781880849571220 us; error 34 us; skew 500 ppm
Jun 19 14:54:09 dist-test-slave-3qdh krb5kdc[842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880849, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:09.573581 1612 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:54:09.574307 1612 webserver.cc:460] Webserver started at http://127.19.34.66:37287/ using document root <none> and password file <none>
I20260619 14:54:09.574772 1612 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:09.574832 1612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:09.578167 1612 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:54:09.579782 1633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:09.580812 1612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:54:09.580889 1612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9"
format_stamp: "Formatted at 2026-06-19 14:54:08 on dist-test-slave-3qdh"
server_key: "04959784ced27dadcc740abc2c93b07a"
server_key_iv: "b563053cd03271b1a36da10f13a9a659"
server_key_version: "encryptionkey@0"
I20260619 14:54:09.581295 1612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:09.602152 1612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:09.604549 1612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:09.604678 1612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:09.605135 1612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:09.605842 1612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:09.605886 1612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:09.605923 1612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:09.605944 1612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:09.613827 1612 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:39079
I20260619 14:54:09.613850 1746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:39079 every 8 connection(s)
I20260619 14:54:09.614634 1612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:54:09.618717 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 1612
I20260619 14:54:09.619097 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:44171
--local_ip_for_outbound_sockets=127.19.34.67
--tserver_master_addrs=127.19.34.126:45351
--webserver_port=44501
--webserver_interface=127.19.34.67
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:45559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:54:09 dist-test-slave-3qdh krb5kdc[842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880849, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:54:09.627342 1613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.616198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:48771 (local address 127.19.34.126:45351)
0619 14:54:09.616358 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:09.616364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:09.617164 (+ 800us) server_negotiation.cc:407] Connection header received
0619 14:54:09.618472 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:09.618475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:09.618530 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:09.618595 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:09.619789 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.620284 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.621370 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.621595 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.621695 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:09.623839 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:09.623842 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:09.623844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:09.623868 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:09.625296 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:09.625844 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:09.625848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:09.625850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:09.625932 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:09.626266 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:09.626271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:09.626272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:09.626506 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:09.626619 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:09.627061 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 14:54:09.627183 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:54:09.628075 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.616604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.66:48771)
0619 14:54:09.617026 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:09.617062 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:09.618284 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:09.618780 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:09.618791 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:09.619156 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:09.619651 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:09.619664 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.620423 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.620427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:09.621253 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:09.621262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.621735 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.621739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:09.621884 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:09.622421 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:09.622434 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:09.623663 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:09.625403 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:09.625408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:09.625422 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:09.625742 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:09.626018 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:09.626022 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:09.626024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:09.626158 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:09.626617 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:09.626623 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:09.626846 (+ 223us) client_negotiation.cc:769] Sending connection context
0619 14:54:09.627009 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 14:54:09.627299 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":99,"threads_started":1}
I20260619 14:54:09.629202 1747 heartbeater.cc:347] Connected to a master server at 127.19.34.126:45351
I20260619 14:54:09.629412 1747 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:09.629839 1747 heartbeater.cc:511] Master 127.19.34.126:45351 requested a full tablet report, sending...
I20260619 14:54:09.630807 1421 ts_manager.cc:194] Registered new tserver with Master: 530f0dc8c0ee46bcac6b01f0ee984dd9 (127.19.34.66:39079)
I20260619 14:54:09.631351 1421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:48771
W20260619 14:54:09.718315 1750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:09.718528 1750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:09.718586 1750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:09.722466 1750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:09.722532 1750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:09.722613 1750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:54:09.727247 1750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:45559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:44171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=44501
--tserver_master_addrs=127.19.34.126:45351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:09.728065 1750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:09.728967 1750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:09.737105 1756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:09.737142 1758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:09.737150 1755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:09.737270 1750 server_base.cc:1034] running on GCE node
I20260619 14:54:09.737671 1750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:09.738221 1750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:09.739377 1750 hybrid_clock.cc:648] HybridClock initialized: now 1781880849739347 us; error 41 us; skew 500 ppm
Jun 19 14:54:09 dist-test-slave-3qdh krb5kdc[842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880849, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:09.741621 1750 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:54:09.742332 1750 webserver.cc:460] Webserver started at http://127.19.34.67:44501/ using document root <none> and password file <none>
I20260619 14:54:09.742856 1750 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:09.742944 1750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:09.746448 1750 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260619 14:54:09.748215 1765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:09.749317 1750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:54:09.749429 1750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "e1d41dd2c3174f229235816f3c80030c"
format_stamp: "Formatted at 2026-06-19 14:54:08 on dist-test-slave-3qdh"
server_key: "ba95c513caaed9ed0a0c1ae3d3aebfae"
server_key_iv: "0084569e94c1a47488df92d67090fdd5"
server_key_version: "encryptionkey@0"
I20260619 14:54:09.749861 1750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:09.787935 1750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:09.790517 1750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:09.790692 1750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:09.791309 1750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:09.792236 1750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:09.792295 1750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:09.792374 1750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:09.792404 1750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:09.801263 1750 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:44171
I20260619 14:54:09.801296 1878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:44171 every 8 connection(s)
I20260619 14:54:09.802155 1750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:54:09.804584 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 1750
Jun 19 14:54:09 dist-test-slave-3qdh krb5kdc[842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880849, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:54:09.813349 1882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.805211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45764 (local address 127.19.34.126:45351)
0619 14:54:09.805433 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:09.805437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:09.805614 (+ 177us) server_negotiation.cc:407] Connection header received
0619 14:54:09.805761 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:09.805764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:09.805806 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:09.805897 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:09.806899 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.807397 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.808331 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.808542 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.808623 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:09.810046 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:09.810049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:09.810050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:09.810108 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:09.811609 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:09.812117 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:09.812119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:09.812120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:09.812187 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:09.812443 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:09.812446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:09.812447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:09.812726 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:09.812933 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:09.813066 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 14:54:09.813186 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"spinlock_wait_cycles":2304,"thread_start_us":70,"threads_started":1}
I20260619 14:54:09.815761 1613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.803607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:48743 (local address 127.19.34.126:45351)
0619 14:54:09.803736 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:09.803741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:09.804606 (+ 865us) server_negotiation.cc:407] Connection header received
0619 14:54:09.806334 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:09.806336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:09.806367 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:09.806437 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:09.807643 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.808092 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.809449 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.809721 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.809800 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:09.812425 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:09.812430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:09.812434 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:09.812458 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:09.813982 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:09.814457 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:09.814462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:09.814463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:09.814533 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:09.814889 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:09.814891 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:09.814892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:09.815097 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:09.815176 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:09.815572 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 14:54:09.815658 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:54:09.816531 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.804002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.67:48743)
0619 14:54:09.804454 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:09.804505 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:09.806119 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:09.806694 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:09.806702 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:09.807058 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:09.807501 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:09.807519 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.808205 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.808208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:09.809338 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:09.809348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.809872 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.809899 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:09.810119 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:09.810724 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:09.810737 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:09.812210 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:09.814120 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:09.814128 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:09.814141 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:09.814364 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:09.814633 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:09.814636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:09.814638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:09.814770 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:09.815184 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:09.815189 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:09.815367 (+ 178us) client_negotiation.cc:769] Sending connection context
0619 14:54:09.815519 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 14:54:09.815730 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":97,"threads_started":1}
I20260619 14:54:09.817551 1879 heartbeater.cc:347] Connected to a master server at 127.19.34.126:45351
I20260619 14:54:09.817744 1879 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:09.818181 1879 heartbeater.cc:511] Master 127.19.34.126:45351 requested a full tablet report, sending...
I20260619 14:54:09.819101 1421 ts_manager.cc:194] Registered new tserver with Master: e1d41dd2c3174f229235816f3c80030c (127.19.34.67:44171)
I20260619 14:54:09.819583 1421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:48743
I20260619 14:54:09.826360 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:54:09.829700 1613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.826896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45766 (local address 127.19.34.126:45351)
0619 14:54:09.827009 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:09.827012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:09.827112 (+ 100us) server_negotiation.cc:407] Connection header received
0619 14:54:09.827212 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:09.827213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:09.827284 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:09.827349 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:54:09.827484 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.827896 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.828672 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.828846 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.828914 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:09.829155 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:54:09.829418 (+ 263us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:54:09.829489 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:09.829550 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 14:54:09.829582 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:54:09.830224 1421 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:54:09.833158 1421 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:54:09.839954 1543 tablet_service.cc:1467] Processing CreateTablet for tablet c72c55ee68184db7b93834b9fd191e15 (DEFAULT_TABLE table=test-table [id=2893615074474578814c687f506f477d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:09.840591 1543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c72c55ee68184db7b93834b9fd191e15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:09.843284 1501 tablet_bootstrap.cc:492] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d: Bootstrap starting.
I20260619 14:54:09.844066 1501 tablet_bootstrap.cc:654] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:09.844211 1616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.839007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:39079 (local address 127.0.0.1:60626)
0619 14:54:09.839332 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:09.839350 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:09.839432 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:09.840187 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:09.840215 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:09.840230 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:09.840299 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:09.840305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.841540 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.841546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:09.842634 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:09.842643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.843780 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.843783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:09.843860 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:09.843898 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 14:54:09.843977 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 14:54:09.844090 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":39}
I20260619 14:54:09.844324 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.839216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.19.34.66:39079)
0619 14:54:09.839656 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:09.839675 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:09.839699 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:54:09.839780 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:09.839789 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:09.840005 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:09.840214 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:09.840453 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.841394 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.842787 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.843648 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.843765 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:09.843924 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:09.844029 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 14:54:09.844132 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":118,"threads_started":1}
I20260619 14:54:09.844514 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.839233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:44171 (local address 127.0.0.1:58696)
0619 14:54:09.839604 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:09.839618 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:09.839695 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:09.840542 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:09.840545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:09.840569 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:09.840666 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:09.840674 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.841662 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.841665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:09.842770 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:09.842781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.843890 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.843894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:09.843986 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:09.844026 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 14:54:09.844077 (+ 51us) client_negotiation.cc:240] Negotiation successful
0619 14:54:09.844140 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":43,"thread_start_us":70,"threads_started":1}
I20260619 14:54:09.845101 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:09.839541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58696 (local address 127.19.34.67:44171)
0619 14:54:09.840082 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:09.840097 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:09.840121 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:54:09.840226 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:09.840231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:09.840404 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:09.840558 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:09.840802 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.841485 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.842908 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:09.843755 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:09.844700 (+ 945us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:09.844822 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:09.844866 (+ 44us) server_negotiation.cc:299] Negotiation successful
0619 14:54:09.844958 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":106,"threads_started":1}
I20260619 14:54:09.845742 1501 tablet_bootstrap.cc:492] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d: No bootstrap required, opened a new log
I20260619 14:54:09.845829 1501 ts_tablet_manager.cc:1397] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:54:09.846213 1501 raft_consensus.cc:348] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:09.846324 1501 raft_consensus.cc:374] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:09.846367 1501 raft_consensus.cc:729] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd5de741a6004d7cb5132cbe2e86365d, State: Initialized, Role: FOLLOWER
I20260619 14:54:09.846480 1501 consensus_queue.cc:260] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:09.846751 1501 ts_tablet_manager.cc:1428] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:54:09.847119 1681 tablet_service.cc:1467] Processing CreateTablet for tablet c72c55ee68184db7b93834b9fd191e15 (DEFAULT_TABLE table=test-table [id=2893615074474578814c687f506f477d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:09.847781 1813 tablet_service.cc:1467] Processing CreateTablet for tablet c72c55ee68184db7b93834b9fd191e15 (DEFAULT_TABLE table=test-table [id=2893615074474578814c687f506f477d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:09.848198 1681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c72c55ee68184db7b93834b9fd191e15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:09.848752 1813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c72c55ee68184db7b93834b9fd191e15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:09.855472 1893 tablet_bootstrap.cc:492] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c: Bootstrap starting.
I20260619 14:54:09.856441 1892 tablet_bootstrap.cc:492] T c72c55ee68184db7b93834b9fd191e15 P 530f0dc8c0ee46bcac6b01f0ee984dd9: Bootstrap starting.
I20260619 14:54:09.857436 1893 tablet_bootstrap.cc:654] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:09.858389 1893 log.cc:824] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:09.858479 1892 tablet_bootstrap.cc:654] T c72c55ee68184db7b93834b9fd191e15 P 530f0dc8c0ee46bcac6b01f0ee984dd9: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:09.859210 1892 log.cc:824] T c72c55ee68184db7b93834b9fd191e15 P 530f0dc8c0ee46bcac6b01f0ee984dd9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:09.860105 1893 tablet_bootstrap.cc:492] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c: No bootstrap required, opened a new log
I20260619 14:54:09.860273 1893 ts_tablet_manager.cc:1397] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:54:09.860786 1892 tablet_bootstrap.cc:492] T c72c55ee68184db7b93834b9fd191e15 P 530f0dc8c0ee46bcac6b01f0ee984dd9: No bootstrap required, opened a new log
I20260619 14:54:09.860972 1892 ts_tablet_manager.cc:1397] T c72c55ee68184db7b93834b9fd191e15 P 530f0dc8c0ee46bcac6b01f0ee984dd9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260619 14:54:09.863070 1893 raft_consensus.cc:348] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:09.863323 1893 raft_consensus.cc:374] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:09.863390 1893 raft_consensus.cc:729] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1d41dd2c3174f229235816f3c80030c, State: Initialized, Role: FOLLOWER
I20260619 14:54:09.863757 1893 consensus_queue.cc:260] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:09.863888 1892 raft_consensus.cc:348] T c72c55ee68184db7b93834b9fd191e15 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:09.864152 1892 raft_consensus.cc:374] T c72c55ee68184db7b93834b9fd191e15 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:09.864210 1892 raft_consensus.cc:729] T c72c55ee68184db7b93834b9fd191e15 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 530f0dc8c0ee46bcac6b01f0ee984dd9, State: Initialized, Role: FOLLOWER
I20260619 14:54:09.864385 1879 heartbeater.cc:503] Master 127.19.34.126:45351 was elected leader, sending a full tablet report...
I20260619 14:54:09.864629 1893 ts_tablet_manager.cc:1428] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 14:54:09.864622 1892 consensus_queue.cc:260] T c72c55ee68184db7b93834b9fd191e15 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:09.865156 1747 heartbeater.cc:503] Master 127.19.34.126:45351 was elected leader, sending a full tablet report...
I20260619 14:54:09.865520 1892 ts_tablet_manager.cc:1428] T c72c55ee68184db7b93834b9fd191e15 P 530f0dc8c0ee46bcac6b01f0ee984dd9: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260619 14:54:09.941068 1610 tablet.cc:2369] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:54:10.037288 1896 raft_consensus.cc:482] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:54:10.037529 1896 raft_consensus.cc:504] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:10.038630 1896 leader_election.cc:288] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 530f0dc8c0ee46bcac6b01f0ee984dd9 (127.19.34.66:39079), bd5de741a6004d7cb5132cbe2e86365d (127.19.34.65:44209)
I20260619 14:54:10.044425 1889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:10.039269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:43449 (local address 127.19.34.66:39079)
0619 14:54:10.039446 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:10.039453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:10.039467 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:54:10.039517 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:10.039522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:10.039596 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:10.039662 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:10.040006 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:10.041602 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:10.043307 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:10.043927 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:10.044054 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:10.044114 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:10.044238 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 14:54:10.044316 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 14:54:10.044435 1898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:10.038837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:39079 (local address 127.19.34.67:43449)
0619 14:54:10.039245 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:10.039254 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:10.039347 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:10.039685 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:10.039688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:10.039705 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:10.039858 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:10.039867 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:10.041715 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:10.041719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:10.043187 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:10.043201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:10.044046 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:10.044049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:10.044135 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:10.044157 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:54:10.044221 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 14:54:10.044283 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":28,"thread_start_us":108,"threads_started":1}
I20260619 14:54:10.044729 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:10.038817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:44209 (local address 127.19.34.67:54885)
0619 14:54:10.038974 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:10.038990 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:10.039073 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:10.040084 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:10.040087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:10.040103 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:10.040184 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:10.040189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:10.041451 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:10.041457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:10.042302 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:10.042311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:10.044352 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:10.044358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:10.044461 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:10.044487 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:54:10.044534 (+ 47us) client_negotiation.cc:240] Negotiation successful
0619 14:54:10.044596 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":1}
I20260619 14:54:10.045624 1701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c72c55ee68184db7b93834b9fd191e15" candidate_uuid: "e1d41dd2c3174f229235816f3c80030c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" is_pre_election: true
I20260619 14:54:10.045737 1899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:10.038912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:54885 (local address 127.19.34.65:44209)
0619 14:54:10.039559 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:10.039565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:10.039815 (+ 250us) server_negotiation.cc:407] Connection header received
0619 14:54:10.039874 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:10.039877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:10.039949 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:10.040076 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:10.040286 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:10.041287 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:10.043344 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:10.044200 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:10.045338 (+ 1138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:10.045497 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:10.045533 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 14:54:10.045599 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":73,"threads_started":1}
I20260619 14:54:10.045931 1701 raft_consensus.cc:2454] T c72c55ee68184db7b93834b9fd191e15 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1d41dd2c3174f229235816f3c80030c in term 0.
I20260619 14:54:10.046241 1563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c72c55ee68184db7b93834b9fd191e15" candidate_uuid: "e1d41dd2c3174f229235816f3c80030c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd5de741a6004d7cb5132cbe2e86365d" is_pre_election: true
I20260619 14:54:10.046397 1768 leader_election.cc:302] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530f0dc8c0ee46bcac6b01f0ee984dd9, e1d41dd2c3174f229235816f3c80030c; no voters:
I20260619 14:54:10.046521 1563 raft_consensus.cc:2454] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1d41dd2c3174f229235816f3c80030c in term 0.
I20260619 14:54:10.046693 1896 raft_consensus.cc:2781] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:54:10.046770 1896 raft_consensus.cc:482] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:54:10.046811 1896 raft_consensus.cc:3037] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:10.047753 1896 raft_consensus.cc:504] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:10.048126 1896 leader_election.cc:288] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c [CANDIDATE]: Term 1 election: Requested vote from peers 530f0dc8c0ee46bcac6b01f0ee984dd9 (127.19.34.66:39079), bd5de741a6004d7cb5132cbe2e86365d (127.19.34.65:44209)
I20260619 14:54:10.048409 1701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c72c55ee68184db7b93834b9fd191e15" candidate_uuid: "e1d41dd2c3174f229235816f3c80030c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9"
I20260619 14:54:10.048535 1701 raft_consensus.cc:3037] T c72c55ee68184db7b93834b9fd191e15 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:10.048547 1563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c72c55ee68184db7b93834b9fd191e15" candidate_uuid: "e1d41dd2c3174f229235816f3c80030c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd5de741a6004d7cb5132cbe2e86365d"
I20260619 14:54:10.048663 1563 raft_consensus.cc:3037] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:10.049479 1701 raft_consensus.cc:2454] T c72c55ee68184db7b93834b9fd191e15 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1d41dd2c3174f229235816f3c80030c in term 1.
I20260619 14:54:10.049518 1563 raft_consensus.cc:2454] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1d41dd2c3174f229235816f3c80030c in term 1.
I20260619 14:54:10.049811 1768 leader_election.cc:302] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 530f0dc8c0ee46bcac6b01f0ee984dd9, e1d41dd2c3174f229235816f3c80030c; no voters:
I20260619 14:54:10.049991 1896 raft_consensus.cc:2781] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:10.050207 1896 raft_consensus.cc:686] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c [term 1 LEADER]: Becoming Leader. State: Replica: e1d41dd2c3174f229235816f3c80030c, State: Running, Role: LEADER
I20260619 14:54:10.050522 1896 consensus_queue.cc:237] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
W20260619 14:54:10.052959 1880 tablet.cc:2369] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:54:10.053711 1421 catalog_manager.cc:5495] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c reported cstate change: term changed from 0 to 1, leader changed from <none> to e1d41dd2c3174f229235816f3c80030c (127.19.34.67). New cstate: current_term: 1 leader_uuid: "e1d41dd2c3174f229235816f3c80030c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } health_report { overall_health: UNKNOWN } } }
I20260619 14:54:10.079110 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:10.075139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.19.34.67:44171)
0619 14:54:10.075292 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:10.075296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:10.075308 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:54:10.075456 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:10.075458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:10.075521 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:10.075622 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:54:10.075844 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:10.076589 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:10.077712 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:10.078001 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:10.078099 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:10.078362 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:54:10.078785 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:54:10.078882 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:10.078934 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 14:54:10.078976 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 14:54:10.085150 1563 raft_consensus.cc:1261] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Refusing update from remote peer e1d41dd2c3174f229235816f3c80030c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:10.085772 1896 consensus_queue.cc:1035] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:10.087378 1701 raft_consensus.cc:1261] T c72c55ee68184db7b93834b9fd191e15 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 1 FOLLOWER]: Refusing update from remote peer e1d41dd2c3174f229235816f3c80030c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:10.087967 1896 consensus_queue.cc:1035] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c [LEADER]: Connected to new peer: Peer: permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:10.092402 1903 mvcc.cc:205] Tried to move back new op lower bound from 7298583961942618112 to 7298583961809842176. Current Snapshot: MvccSnapshot[applied={T|T < 7298583961942618112}]
I20260619 14:54:10.100351 1420 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:10.100490 1420 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:10.102418 1420 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5: Sending DeleteTablet for 3 replicas of tablet c72c55ee68184db7b93834b9fd191e15
I20260619 14:54:10.102938 1681 tablet_service.cc:1514] Processing DeleteTablet for tablet c72c55ee68184db7b93834b9fd191e15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:10 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:60626
I20260619 14:54:10.102996 1813 tablet_service.cc:1514] Processing DeleteTablet for tablet c72c55ee68184db7b93834b9fd191e15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:10 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:58696
I20260619 14:54:10.103140 1543 tablet_service.cc:1514] Processing DeleteTablet for tablet c72c55ee68184db7b93834b9fd191e15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:10 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:42512
I20260619 14:54:10.103430 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 1459
I20260619 14:54:10.103488 1914 tablet_replica.cc:330] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c: stopping tablet replica
I20260619 14:54:10.103739 1914 raft_consensus.cc:2229] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:54:10.104140 1914 raft_consensus.cc:2258] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:10.105152 1913 tablet_replica.cc:330] T c72c55ee68184db7b93834b9fd191e15 P 530f0dc8c0ee46bcac6b01f0ee984dd9: stopping tablet replica
I20260619 14:54:10.105468 1913 raft_consensus.cc:2229] T c72c55ee68184db7b93834b9fd191e15 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:54:10.105545 1914 ts_tablet_manager.cc:1905] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:10.105850 1913 raft_consensus.cc:2258] T c72c55ee68184db7b93834b9fd191e15 P 530f0dc8c0ee46bcac6b01f0ee984dd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:10.107097 1913 ts_tablet_manager.cc:1905] T c72c55ee68184db7b93834b9fd191e15 P 530f0dc8c0ee46bcac6b01f0ee984dd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:10.108587 1914 ts_tablet_manager.cc:1918] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:10.108691 1914 log.cc:1196] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/c72c55ee68184db7b93834b9fd191e15
I20260619 14:54:10.109033 1914 ts_tablet_manager.cc:1939] T c72c55ee68184db7b93834b9fd191e15 P e1d41dd2c3174f229235816f3c80030c: Deleting consensus metadata
I20260619 14:54:10.110036 1407 catalog_manager.cc:4841] TS e1d41dd2c3174f229235816f3c80030c (127.19.34.67:44171): tablet c72c55ee68184db7b93834b9fd191e15 (table test-table [id=2893615074474578814c687f506f477d]) successfully deleted
I20260619 14:54:10.110694 1913 ts_tablet_manager.cc:1918] T c72c55ee68184db7b93834b9fd191e15 P 530f0dc8c0ee46bcac6b01f0ee984dd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:10.110826 1913 log.cc:1196] T c72c55ee68184db7b93834b9fd191e15 P 530f0dc8c0ee46bcac6b01f0ee984dd9: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/wal/wals/c72c55ee68184db7b93834b9fd191e15
I20260619 14:54:10.111194 1913 ts_tablet_manager.cc:1939] T c72c55ee68184db7b93834b9fd191e15 P 530f0dc8c0ee46bcac6b01f0ee984dd9: Deleting consensus metadata
W20260619 14:54:10.111212 1405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.34.65:44209 (error 108)
I20260619 14:54:10.111714 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 1612
I20260619 14:54:10.117131 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:10.116795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:44209 (local address 127.0.0.1:42516)
0619 14:54:10.116975 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:10.117043 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.65:44209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 14:54:10.117154 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 1750
I20260619 14:54:10.117779 1616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:10.117207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:39079 (local address 127.0.0.1:60628)
0619 14:54:10.117675 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:10.117719 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.66:39079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377}
W20260619 14:54:10.118029 1405 catalog_manager.cc:4584] TS bd5de741a6004d7cb5132cbe2e86365d (127.19.34.65:44209): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c72c55ee68184db7b93834b9fd191e15: Network error: Client connection negotiation failed: client connection to 127.19.34.65:44209: connect: Connection refused (error 111)
I20260619 14:54:10.122249 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 1388
I20260619 14:54:10.127736 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:45351
--webserver_interface=127.19.34.126
--webserver_port=38623
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:45559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:45351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:10.224076 1917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:10.224326 1917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:10.224381 1917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:10.228030 1917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:10.228098 1917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:10.228132 1917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:10.228178 1917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:10.228209 1917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:10.232652 1917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:45559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:45351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:45351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=38623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:10.233541 1917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:10.234377 1917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:10.241979 1922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:10.241988 1925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:10.242067 1923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:10.242288 1917 server_base.cc:1034] running on GCE node
I20260619 14:54:10.242779 1917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:10.243695 1917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:10.244856 1917 hybrid_clock.cc:648] HybridClock initialized: now 1781880850244847 us; error 29 us; skew 500 ppm
Jun 19 14:54:10 dist-test-slave-3qdh krb5kdc[842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880850, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:10.247364 1917 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:54:10.248028 1917 webserver.cc:460] Webserver started at http://127.19.34.126:38623/ using document root <none> and password file <none>
I20260619 14:54:10.248538 1917 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:10.248600 1917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:10.251844 1917 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:54:10.253751 1932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:10.254812 1917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:54:10.254895 1917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "5fda20c76804435e88c1573a73fc0ab5"
format_stamp: "Formatted at 2026-06-19 14:54:08 on dist-test-slave-3qdh"
server_key: "da42ef64eca699ce16385e16099e1951"
server_key_iv: "fe4227969ffa66f60a48443485ab6221"
server_key_version: "encryptionkey@0"
I20260619 14:54:10.255329 1917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:10.266546 1917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:10.269166 1917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:10.269285 1917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:10.274998 1917 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:45351
I20260619 14:54:10.275285 1984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:45351 every 8 connection(s)
I20260619 14:54:10.276010 1917 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:10.279728 1985 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:54:10.281258 1985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5: Bootstrap starting.
I20260619 14:54:10.282939 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 1917
I20260619 14:54:10.283345 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:44209
--local_ip_for_outbound_sockets=127.19.34.65
--tserver_master_addrs=127.19.34.126:45351
--webserver_port=34327
--webserver_interface=127.19.34.65
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:45559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:54:10.296882 1985 log.cc:824] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:10.306229 1985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:54:10.306557 1985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5: Bootstrap complete.
I20260619 14:54:10.310052 1985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fda20c76804435e88c1573a73fc0ab5" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45351 } }
I20260619 14:54:10.310372 1985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fda20c76804435e88c1573a73fc0ab5, State: Initialized, Role: FOLLOWER
I20260619 14:54:10.310774 1985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fda20c76804435e88c1573a73fc0ab5" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45351 } }
I20260619 14:54:10.310868 1985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:10.310940 1985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:10.311038 1985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 2 FOLLOWER]: Advancing to term 3
I20260619 14:54:10.312668 1985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fda20c76804435e88c1573a73fc0ab5" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45351 } }
I20260619 14:54:10.312937 1985 leader_election.cc:302] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fda20c76804435e88c1573a73fc0ab5; no voters:
I20260619 14:54:10.313201 1985 leader_election.cc:288] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 14:54:10.313261 1989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 14:54:10.313441 1989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [term 3 LEADER]: Becoming Leader. State: Replica: 5fda20c76804435e88c1573a73fc0ab5, State: Running, Role: LEADER
I20260619 14:54:10.313675 1989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fda20c76804435e88c1573a73fc0ab5" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45351 } }
I20260619 14:54:10.314203 1985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:10.314435 1991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5fda20c76804435e88c1573a73fc0ab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fda20c76804435e88c1573a73fc0ab5" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45351 } } }
I20260619 14:54:10.314643 1991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:10.314446 1989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fda20c76804435e88c1573a73fc0ab5. Latest consensus state: current_term: 3 leader_uuid: "5fda20c76804435e88c1573a73fc0ab5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fda20c76804435e88c1573a73fc0ab5" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 45351 } } }
I20260619 14:54:10.314914 1989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fda20c76804435e88c1573a73fc0ab5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:10.315152 1993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:10.319983 1993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:10.320708 1993 catalog_manager.cc:1259] Loaded cluster ID: 3683abfa882f484c8883630a35127119
I20260619 14:54:10.320768 1993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 14:54:10.321651 1993 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa0ea42b9ad google::LogMessage::Fail() at ??:0
@ 0x7fa0ea42e88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa0ea42b4de google::LogMessage::Flush() at ??:0
@ 0x7fa0ea42ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa0f4d6afdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fa0f4d6bd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa0f4d68579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fa0f4dac640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa0ec67d3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa0ec67dc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fa0ec67f3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa0ec66e512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa0edcb26db start_thread at ??:0
@ 0x7fa0e8f2961f clone at ??:0
W20260619 14:54:10.398716 1987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:10.398953 1987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:10.399065 1987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:10.402595 1987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:10.402667 1987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:10.402771 1987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:54:10.407311 1987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:45559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:44209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=34327
--tserver_master_addrs=127.19.34.126:45351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:10.408180 1987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:10.409145 1987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:10.417670 2027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:10.417652 2026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:10.417646 2029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:10.418321 1987 server_base.cc:1034] running on GCE node
I20260619 14:54:10.418704 1987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:10.419325 1987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:10.420485 1987 hybrid_clock.cc:648] HybridClock initialized: now 1781880850420458 us; error 44 us; skew 500 ppm
Jun 19 14:54:10 dist-test-slave-3qdh krb5kdc[842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880850, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:10.422891 1987 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:54:10.423678 1987 webserver.cc:460] Webserver started at http://127.19.34.65:34327/ using document root <none> and password file <none>
I20260619 14:54:10.424172 1987 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:10.424230 1987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:10.427826 1987 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:54:10.430035 2036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:10.431217 1987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:54:10.431350 1987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "bd5de741a6004d7cb5132cbe2e86365d"
format_stamp: "Formatted at 2026-06-19 14:54:08 on dist-test-slave-3qdh"
server_key: "bb42d1c75047b6999106dd06f2602935"
server_key_iv: "3f28884ec4939ad766f43d8358741d5c"
server_key_version: "encryptionkey@0"
I20260619 14:54:10.431797 1987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:10.443274 1987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:10.446089 1987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:10.446281 1987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:10.446830 1987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:10.447753 2043 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:54:10.452337 1987 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:54:10.452437 1987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 14:54:10.452508 1987 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:54:10.453867 1987 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:54:10.453930 1987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 14:54:10.454187 2043 tablet_bootstrap.cc:492] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d: Bootstrap starting.
I20260619 14:54:10.464196 1987 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:44209
I20260619 14:54:10.464264 2150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:44209 every 8 connection(s)
I20260619 14:54:10.465042 1987 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:54:10.469269 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 1987
I20260619 14:54:10.469624 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:39079
--local_ip_for_outbound_sockets=127.19.34.66
--tserver_master_addrs=127.19.34.126:45351
--webserver_port=37287
--webserver_interface=127.19.34.66
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:45559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:54:10.474918 2043 log.cc:824] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:10.481333 2043 tablet_bootstrap.cc:492] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:54:10.481659 2043 tablet_bootstrap.cc:492] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d: Bootstrap complete.
I20260619 14:54:10.482583 2043 ts_tablet_manager.cc:1397] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d: Time spent bootstrapping tablet: real 0.029s user 0.020s sys 0.004s
I20260619 14:54:10.484666 2043 raft_consensus.cc:348] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:10.485023 2043 raft_consensus.cc:729] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd5de741a6004d7cb5132cbe2e86365d, State: Initialized, Role: FOLLOWER
I20260619 14:54:10.485435 2043 consensus_queue.cc:260] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:10.486335 2043 ts_tablet_manager.cc:1428] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260619 14:54:10.524215 2153 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:54:10.466844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.65:42485)
0619 14:54:10.467269 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:10.467317 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:10.468066 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:10.523558 (+ 55492us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":123,"threads_started":1}
W20260619 14:54:10.524720 2151 heartbeater.cc:644] Failed to heartbeat to 127.19.34.126:45351 (0 consecutive failures): Network error: Failed to ping master at 127.19.34.126:45351: Client connection negotiation failed: client connection to 127.19.34.126:45351: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:54:10.525369 2153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:10.525149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.65:60207)
0619 14:54:10.525261 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:10.525320 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 14:54:10.525985 2153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:10.525800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.65:36259)
0619 14:54:10.525895 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:10.525939 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260619 14:54:10.526213 2151 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:54:10.582224 2154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:10.582412 2154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:10.582487 2154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:10.586114 2154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:10.586166 2154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:10.586261 2154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:54:10.590623 2154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:45559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:39079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=37287
--tserver_master_addrs=127.19.34.126:45351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:10.591410 2154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:10.592211 2154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:54:10.600203 2154 server_base.cc:1034] running on GCE node
W20260619 14:54:10.600160 2164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:10.600138 2162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:10.600153 2161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:10.600699 2154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:10.601207 2154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:10.602362 2154 hybrid_clock.cc:648] HybridClock initialized: now 1781880850602344 us; error 37 us; skew 500 ppm
Jun 19 14:54:10 dist-test-slave-3qdh krb5kdc[842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880850, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:10.604528 2154 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:54:10.605263 2154 webserver.cc:460] Webserver started at http://127.19.34.66:37287/ using document root <none> and password file <none>
I20260619 14:54:10.605741 2154 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:10.605821 2154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:10.609091 2154 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:54:10.610877 2171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:10.611907 2154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:54:10.612015 2154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9"
format_stamp: "Formatted at 2026-06-19 14:54:08 on dist-test-slave-3qdh"
server_key: "04959784ced27dadcc740abc2c93b07a"
server_key_iv: "b563053cd03271b1a36da10f13a9a659"
server_key_version: "encryptionkey@0"
I20260619 14:54:10.612498 2154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:10.622076 2154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:10.624236 2154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:10.624418 2154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:10.624876 2154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:10.625628 2154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:10.625692 2154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:10.625752 2154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:10.625797 2154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:10.633376 2154 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:39079
I20260619 14:54:10.633405 2284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:39079 every 8 connection(s)
I20260619 14:54:10.634362 2154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:54:10.635892 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 2154
I20260619 14:54:10.636390 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:44171
--local_ip_for_outbound_sockets=127.19.34.67
--tserver_master_addrs=127.19.34.126:45351
--webserver_port=44501
--webserver_interface=127.19.34.67
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:45559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:54:10.638551 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:10.636585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.66:53899)
0619 14:54:10.637056 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:10.637177 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":132,"threads_started":1}
W20260619 14:54:10.641067 2285 heartbeater.cc:644] Failed to heartbeat to 127.19.34.126:45351 (0 consecutive failures): Network error: Failed to ping master at 127.19.34.126:45351: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
I20260619 14:54:10.641661 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:10.641449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.66:53421)
0619 14:54:10.641543 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:10.641592 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 14:54:10.642400 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:10.642181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.66:37947)
0619 14:54:10.642288 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:10.642333 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 14:54:10.642966 2285 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:54:10.746699 2288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:10.746937 2288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:10.747006 2288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:10.750542 2288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:10.750612 2288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:10.750685 2288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:54:10.755313 2288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:45559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:44171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=44501
--tserver_master_addrs=127.19.34.126:45351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:10.756249 2288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:10.757166 2288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:10.765609 2296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:10.765646 2293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:10.765764 2288 server_base.cc:1034] running on GCE node
W20260619 14:54:10.765626 2294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:10.766218 2288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:10.766752 2288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:10.767901 2288 hybrid_clock.cc:648] HybridClock initialized: now 1781880850767877 us; error 36 us; skew 500 ppm
Jun 19 14:54:10 dist-test-slave-3qdh krb5kdc[842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880850, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:10.770612 2288 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:54:10.771378 2288 webserver.cc:460] Webserver started at http://127.19.34.67:44501/ using document root <none> and password file <none>
I20260619 14:54:10.771837 2288 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:10.771888 2288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:10.775199 2288 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:54:10.777194 2303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:10.778239 2288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:54:10.778352 2288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "e1d41dd2c3174f229235816f3c80030c"
format_stamp: "Formatted at 2026-06-19 14:54:08 on dist-test-slave-3qdh"
server_key: "ba95c513caaed9ed0a0c1ae3d3aebfae"
server_key_iv: "0084569e94c1a47488df92d67090fdd5"
server_key_version: "encryptionkey@0"
I20260619 14:54:10.778779 2288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:10.800626 2288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:10.803164 2288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:10.803326 2288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:10.803820 2288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:10.804548 2288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:10.804608 2288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:10.804662 2288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:10.804701 2288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:10.813525 2288 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:44171
I20260619 14:54:10.813542 2416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:44171 every 8 connection(s)
I20260619 14:54:10.814443 2288 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:54:10.818903 2419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:10.816748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.67:41247)
0619 14:54:10.817295 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:10.817490 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":127,"threads_started":1}
W20260619 14:54:10.819365 2417 heartbeater.cc:644] Failed to heartbeat to 127.19.34.126:45351 (0 consecutive failures): Network error: Failed to ping master at 127.19.34.126:45351: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
I20260619 14:54:10.820060 2419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:10.819808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.67:38221)
0619 14:54:10.819943 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:10.820006 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 14:54:10.821139 2419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:10.820960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.67:50911)
0619 14:54:10.821042 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:10.821097 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 14:54:10.821278 2417 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 14:54:10.823266 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 2288
I20260619 14:54:11.527251 2421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:11.526711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.65:60937)
0619 14:54:11.527103 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:11.527164 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":64,"threads_started":1}
I20260619 14:54:11.643769 2422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:11.643412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.66:44095)
0619 14:54:11.643635 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:11.643682 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":71,"threads_started":1}
I20260619 14:54:11.822055 2423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:11.821679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.67:35837)
0619 14:54:11.821879 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:11.821970 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":59,"threads_started":1}
I20260619 14:54:12.214048 2424 raft_consensus.cc:482] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:54:12.214300 2424 raft_consensus.cc:504] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:12.215335 2424 leader_election.cc:288] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 530f0dc8c0ee46bcac6b01f0ee984dd9 (127.19.34.66:39079), e1d41dd2c3174f229235816f3c80030c (127.19.34.67:44171)
Jun 19 14:54:12 dist-test-slave-3qdh krb5kdc[842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880850, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.66@KRBTEST.COM
Jun 19 14:54:12 dist-test-slave-3qdh krb5kdc[842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880850, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.67@KRBTEST.COM
I20260619 14:54:12.227916 2425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:12.215512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:39079 (local address 127.19.34.65:44251)
0619 14:54:12.215761 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:12.215779 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:12.215886 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:12.217663 (+ 1777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:12.217669 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:12.218161 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:12.218563 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:12.218577 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:12.219688 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:12.219691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:12.220484 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:12.220493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:12.221140 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:12.221143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:12.221254 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:12.221865 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:12.221882 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:12.223527 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:12.225652 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:12.225656 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:12.225667 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:12.226038 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:12.226374 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:12.226379 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:12.226381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:12.226515 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:12.227084 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:12.227087 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:12.227283 (+ 196us) client_negotiation.cc:769] Sending connection context
0619 14:54:12.227439 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 14:54:12.227721 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":78,"spinlock_wait_cycles":6912,"thread_start_us":51,"threads_started":1}
I20260619 14:54:12.227926 2426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:12.215642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:44251 (local address 127.19.34.66:39079)
0619 14:54:12.216126 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:12.216143 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:12.216189 (+ 46us) server_negotiation.cc:407] Connection header received
0619 14:54:12.216987 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:12.216993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:12.217308 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:12.217629 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:12.218742 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:12.219568 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:12.220640 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:12.221024 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:12.221179 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:12.223717 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:12.223723 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:12.223736 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:12.223767 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:12.225531 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:12.226154 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:12.226160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:12.226165 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:12.226279 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:12.226614 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:12.226618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:12.226620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:12.226982 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:12.227103 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:12.227541 (+ 438us) server_negotiation.cc:299] Negotiation successful
0619 14:54:12.227750 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":103,"threads_started":1}
I20260619 14:54:12.227967 2428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:12.216020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:39599 (local address 127.19.34.67:44171)
0619 14:54:12.216471 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:12.216490 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:12.216521 (+ 31us) server_negotiation.cc:407] Connection header received
0619 14:54:12.217131 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:12.217135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:12.217399 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:12.217666 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:12.218717 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:12.219524 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:12.220599 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:12.220989 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:12.221146 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:12.224011 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:12.224015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:12.224023 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:12.224045 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:12.225613 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:12.226196 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:12.226204 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:12.226213 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:12.226312 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:12.226611 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:12.226616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:12.226617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:12.226977 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:12.227098 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:12.227541 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 14:54:12.227779 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":111,"threads_started":1}
I20260619 14:54:12.227957 2427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:12.215858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:44171 (local address 127.19.34.65:39599)
0619 14:54:12.216112 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:12.216126 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:12.216261 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:12.217665 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:12.217670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:12.218161 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:12.218570 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:12.218580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:12.219643 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:12.219646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:12.220485 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:12.220493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:12.221095 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:12.221098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:12.221259 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:12.221855 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:12.221873 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:12.223845 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:12.225806 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:12.225810 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:12.225812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:12.226071 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:12.226408 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:12.226411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:12.226413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:12.226519 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:12.227079 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:12.227084 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:12.227286 (+ 202us) client_negotiation.cc:769] Sending connection context
0619 14:54:12.227437 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 14:54:12.227793 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":226,"spinlock_wait_cycles":19712,"thread_start_us":53,"threads_started":1}
W20260619 14:54:12.230548 2039 leader_election.cc:341] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 530f0dc8c0ee46bcac6b01f0ee984dd9 (127.19.34.66:39079): Not found: Tablet not found: c72c55ee68184db7b93834b9fd191e15
W20260619 14:54:12.231027 2039 leader_election.cc:341] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1d41dd2c3174f229235816f3c80030c (127.19.34.67:44171): Not found: Tablet not found: c72c55ee68184db7b93834b9fd191e15
I20260619 14:54:12.231112 2039 leader_election.cc:302] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bd5de741a6004d7cb5132cbe2e86365d; no voters: 530f0dc8c0ee46bcac6b01f0ee984dd9, e1d41dd2c3174f229235816f3c80030c
I20260619 14:54:12.231320 2424 raft_consensus.cc:2726] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:54:12.528177 2427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:12.527878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.65:37117)
0619 14:54:12.528021 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:12.528094 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 14:54:12.644930 2429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:12.644513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.66:38759)
0619 14:54:12.644767 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:12.644823 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
I20260619 14:54:12.823088 2430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:12.822683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.67:38733)
0619 14:54:12.822944 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:12.823007 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":78,"threads_started":1}
I20260619 14:54:13.529265 2431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:13.528840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.65:38751)
0619 14:54:13.529092 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:13.529175 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":67,"threads_started":1}
I20260619 14:54:13.645947 2432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:13.645576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.66:33241)
0619 14:54:13.645801 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:13.645868 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":82,"threads_started":1}
I20260619 14:54:13.824173 2433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:13.823811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.67:50257)
0619 14:54:13.824033 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:13.824081 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":76,"threads_started":1}
I20260619 14:54:14.275316 2434 raft_consensus.cc:482] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:54:14.275457 2434 raft_consensus.cc:504] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:14.275892 2434 leader_election.cc:288] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 530f0dc8c0ee46bcac6b01f0ee984dd9 (127.19.34.66:39079), e1d41dd2c3174f229235816f3c80030c (127.19.34.67:44171)
W20260619 14:54:14.276751 2039 leader_election.cc:341] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1d41dd2c3174f229235816f3c80030c (127.19.34.67:44171): Not found: Tablet not found: c72c55ee68184db7b93834b9fd191e15
W20260619 14:54:14.276921 2039 leader_election.cc:341] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 530f0dc8c0ee46bcac6b01f0ee984dd9 (127.19.34.66:39079): Not found: Tablet not found: c72c55ee68184db7b93834b9fd191e15
I20260619 14:54:14.276995 2039 leader_election.cc:302] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bd5de741a6004d7cb5132cbe2e86365d; no voters: 530f0dc8c0ee46bcac6b01f0ee984dd9, e1d41dd2c3174f229235816f3c80030c
I20260619 14:54:14.277209 2434 raft_consensus.cc:2726] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:54:14.530300 2435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:14.529936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.65:40903)
0619 14:54:14.530158 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:14.530208 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":71,"threads_started":1}
I20260619 14:54:14.647019 2436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:14.646645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.66:51107)
0619 14:54:14.646884 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:14.646934 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":90,"threads_started":1}
I20260619 14:54:14.825160 2437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:14.824816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.67:50721)
0619 14:54:14.825027 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:14.825075 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":67,"threads_started":1}
I20260619 14:54:15.531652 2438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:15.531198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.65:52959)
0619 14:54:15.531499 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:15.531549 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":104,"threads_started":1}
I20260619 14:54:15.648252 2439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:15.647819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.66:34205)
0619 14:54:15.648097 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:15.648160 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
I20260619 14:54:15.826236 2440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:15.825784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.67:36011)
0619 14:54:15.826073 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:15.826134 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":93,"threads_started":1}
I20260619 14:54:16.532713 2441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:16.532335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.65:53193)
0619 14:54:16.532564 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:16.532624 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":65,"threads_started":1}
I20260619 14:54:16.649298 2442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:16.648923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.66:43999)
0619 14:54:16.649167 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:16.649216 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":71,"threads_started":1}
I20260619 14:54:16.827214 2443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:16.826864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.67:35099)
0619 14:54:16.827078 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:16.827125 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":63,"threads_started":1}
I20260619 14:54:17.067658 2444 raft_consensus.cc:482] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:54:17.067775 2444 raft_consensus.cc:504] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:17.068092 2444 leader_election.cc:288] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 530f0dc8c0ee46bcac6b01f0ee984dd9 (127.19.34.66:39079), e1d41dd2c3174f229235816f3c80030c (127.19.34.67:44171)
W20260619 14:54:17.068874 2039 leader_election.cc:341] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 530f0dc8c0ee46bcac6b01f0ee984dd9 (127.19.34.66:39079): Not found: Tablet not found: c72c55ee68184db7b93834b9fd191e15
W20260619 14:54:17.069018 2039 leader_election.cc:341] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1d41dd2c3174f229235816f3c80030c (127.19.34.67:44171): Not found: Tablet not found: c72c55ee68184db7b93834b9fd191e15
I20260619 14:54:17.069056 2039 leader_election.cc:302] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bd5de741a6004d7cb5132cbe2e86365d; no voters: 530f0dc8c0ee46bcac6b01f0ee984dd9, e1d41dd2c3174f229235816f3c80030c
I20260619 14:54:17.069207 2444 raft_consensus.cc:2726] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:54:17.533767 2445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:17.533388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.65:59425)
0619 14:54:17.533613 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:17.533659 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":77,"threads_started":1}
I20260619 14:54:17.650429 2446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:17.650038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.66:35507)
0619 14:54:17.650310 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:17.650356 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":92,"threads_started":1}
I20260619 14:54:17.828258 2447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:17.827865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.67:56003)
0619 14:54:17.828130 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:17.828180 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":63,"threads_started":1}
I20260619 14:54:18.534824 2448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:18.534474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.65:49483)
0619 14:54:18.534696 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:18.534741 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":69,"threads_started":1}
I20260619 14:54:18.651432 2449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:18.651071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.66:39551)
0619 14:54:18.651294 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:18.651348 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":75,"threads_started":1}
I20260619 14:54:18.829253 2450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:18.828897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.67:52485)
0619 14:54:18.829138 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:18.829181 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":102,"threads_started":1}
I20260619 14:54:19.535789 2451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:19.535440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.65:34791)
0619 14:54:19.535661 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:19.535707 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":62,"threads_started":1}
I20260619 14:54:19.652678 2452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:19.652065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.66:47721)
0619 14:54:19.652292 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:19.652598 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":67,"threads_started":1}
I20260619 14:54:19.830246 2453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:19.829900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.67:39995)
0619 14:54:19.830118 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:19.830167 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":67,"threads_started":1}
I20260619 14:54:20.332186 2454 raft_consensus.cc:482] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:54:20.332347 2454 raft_consensus.cc:504] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "530f0dc8c0ee46bcac6b01f0ee984dd9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39079 } } peers { permanent_uuid: "e1d41dd2c3174f229235816f3c80030c" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 44171 } } peers { permanent_uuid: "bd5de741a6004d7cb5132cbe2e86365d" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 44209 } }
I20260619 14:54:20.332752 2454 leader_election.cc:288] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 530f0dc8c0ee46bcac6b01f0ee984dd9 (127.19.34.66:39079), e1d41dd2c3174f229235816f3c80030c (127.19.34.67:44171)
W20260619 14:54:20.333424 2039 leader_election.cc:341] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 530f0dc8c0ee46bcac6b01f0ee984dd9 (127.19.34.66:39079): Not found: Tablet not found: c72c55ee68184db7b93834b9fd191e15
W20260619 14:54:20.333534 2039 leader_election.cc:341] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e1d41dd2c3174f229235816f3c80030c (127.19.34.67:44171): Not found: Tablet not found: c72c55ee68184db7b93834b9fd191e15
I20260619 14:54:20.333562 2039 leader_election.cc:302] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bd5de741a6004d7cb5132cbe2e86365d; no voters: 530f0dc8c0ee46bcac6b01f0ee984dd9, e1d41dd2c3174f229235816f3c80030c
I20260619 14:54:20.333683 2454 raft_consensus.cc:2726] T c72c55ee68184db7b93834b9fd191e15 P bd5de741a6004d7cb5132cbe2e86365d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:54:20.536902 2455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:20.536529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.65:33881)
0619 14:54:20.536761 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:20.536811 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260619 14:54:20.653927 2456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:20.653409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.66:56729)
0619 14:54:20.653706 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:20.653771 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":92,"threads_started":1}
I20260619 14:54:20.831337 2457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:20.830836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.67:51197)
0619 14:54:20.831220 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:20.831267 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":87,"threads_started":1}
I20260619 14:54:21.537977 2458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:21.537592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.65:43617)
0619 14:54:21.537849 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:21.537895 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":88,"threads_started":1}
I20260619 14:54:21.655489 2459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:21.654815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.66:53117)
0619 14:54:21.655338 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:21.655405 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":277,"threads_started":1}
I20260619 14:54:21.832329 2460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:21.831962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.67:53363)
0619 14:54:21.832172 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:21.832216 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":76,"threads_started":1}
I20260619 14:54:22.539076 2461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:22.538706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.65:48691)
0619 14:54:22.538939 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:22.538988 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":62,"threads_started":1}
I20260619 14:54:22.656733 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:22.656251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.66:36539)
0619 14:54:22.656585 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:22.656636 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":100,"threads_started":1}
I20260619 14:54:22.833271 2463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:22.832943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.67:49547)
0619 14:54:22.833140 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:22.833193 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":61,"threads_started":1}
I20260619 14:54:23.540153 2464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:23.539764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.65:42049)
0619 14:54:23.540012 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:23.540063 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
I20260619 14:54:23.657742 2465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:23.657375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.66:55243)
0619 14:54:23.657607 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:23.657658 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":82,"threads_started":1}
I20260619 14:54:23.834256 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:23.833874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.67:57501)
0619 14:54:23.834131 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:23.834178 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":66,"threads_started":1}
I20260619 14:54:24.541275 2467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:24.540886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.65:35359)
0619 14:54:24.541123 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:24.541185 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":68,"threads_started":1}
I20260619 14:54:24.658764 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:24.658427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.66:42471)
0619 14:54:24.658634 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:24.658682 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":66,"threads_started":1}
I20260619 14:54:24.835206 2469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:24.834837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.67:47479)
0619 14:54:24.835075 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:24.835119 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":61,"threads_started":1}
I20260619 14:54:25.542402 2470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:25.542054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.65:37989)
0619 14:54:25.542275 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:25.542318 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":75,"threads_started":1}
I20260619 14:54:25.659909 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:25.659499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.66:36095)
0619 14:54:25.659749 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:25.659802 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
I20260619 14:54:25.828264 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 1987
I20260619 14:54:25.833369 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 2154
I20260619 14:54:25.836401 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:25.835944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:45351 (local address 127.19.34.67:45583)
0619 14:54:25.836186 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:25.836250 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:45351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":59,"threads_started":1}
I20260619 14:54:25.838498 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 2288
2026-06-19T14:54:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18840 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:54:25 dist-test-slave-3qdh krb5kdc[2475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:54:25 dist-test-slave-3qdh krb5kdc[2475](info): set up 2 sockets
Jun 19 14:54:25 dist-test-slave-3qdh krb5kdc[2475](info): commencing operation
krb5kdc: starting...
W20260619 14:54:26.813503 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:54:26 dist-test-slave-3qdh krb5kdc[2475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:54:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:54:26.917474 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:43807
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:43807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:27.016505 2488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:27.016724 2488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:27.016760 2488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:27.020670 2488 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 14:54:27.020714 2488 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 14:54:27.020730 2488 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 14:54:27.020744 2488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:27.020761 2488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:27.020779 2488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:27.020838 2488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:27.020859 2488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:27.025475 2488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:43807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:43807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:27.026305 2488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:27.027236 2488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:27.035354 2494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:27.035333 2493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:27.035373 2488 server_base.cc:1034] running on GCE node
W20260619 14:54:27.035333 2496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:27.036108 2488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:27.037053 2488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:27.038214 2488 hybrid_clock.cc:648] HybridClock initialized: now 1781880867038197 us; error 30 us; skew 500 ppm
Jun 19 14:54:27 dist-test-slave-3qdh krb5kdc[2475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880867, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:27.040700 2488 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:54:27.041455 2488 webserver.cc:460] Webserver started at http://127.19.34.126:35039/ using document root <none> and password file <none>
I20260619 14:54:27.041945 2488 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:27.042029 2488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:27.042243 2488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:27.043995 2488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "fcd5a6df75864d7ea35630d318368cd3"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-3qdh"
server_key: "17ed842b98f4c2e99a1d184bb2730606"
server_key_iv: "5b54375eed258c0ef2c2e011a79df3ba"
server_key_version: "encryptionkey@0"
I20260619 14:54:27.044523 2488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "fcd5a6df75864d7ea35630d318368cd3"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-3qdh"
server_key: "17ed842b98f4c2e99a1d184bb2730606"
server_key_iv: "5b54375eed258c0ef2c2e011a79df3ba"
server_key_version: "encryptionkey@0"
I20260619 14:54:27.047739 2488 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 14:54:27.049926 2503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:27.050908 2488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:54:27.051038 2488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "fcd5a6df75864d7ea35630d318368cd3"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-3qdh"
server_key: "17ed842b98f4c2e99a1d184bb2730606"
server_key_iv: "5b54375eed258c0ef2c2e011a79df3ba"
server_key_version: "encryptionkey@0"
I20260619 14:54:27.051152 2488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:27.078083 2488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:27.080636 2488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:27.080819 2488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:27.087016 2488 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:43807
I20260619 14:54:27.087314 2555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:43807 every 8 connection(s)
I20260619 14:54:27.087920 2488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:27.090251 2556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:27.092453 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 2488
I20260619 14:54:27.092561 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:54:27.092839 19593 external_mini_cluster.cc:1428] Setting key 3dc7ae01b2dee8c3b037326198592c2c
I20260619 14:54:27.095512 2556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: Bootstrap starting.
I20260619 14:54:27.097540 2556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:27.098241 2556 log.cc:824] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:27.099879 2556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: No bootstrap required, opened a new log
Jun 19 14:54:27 dist-test-slave-3qdh krb5kdc[2475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:54:27.102937 2556 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5a6df75864d7ea35630d318368cd3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43807 } }
I20260619 14:54:27.103134 2556 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:27.103197 2556 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd5a6df75864d7ea35630d318368cd3, State: Initialized, Role: FOLLOWER
I20260619 14:54:27.103606 2556 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5a6df75864d7ea35630d318368cd3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43807 } }
I20260619 14:54:27.103745 2556 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:27.103819 2556 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:27.103948 2556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:27.104840 2556 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5a6df75864d7ea35630d318368cd3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43807 } }
I20260619 14:54:27.105158 2556 leader_election.cc:302] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcd5a6df75864d7ea35630d318368cd3; no voters:
I20260619 14:54:27.105451 2556 leader_election.cc:288] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:54:27.105540 2561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:27.105758 2561 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 1 LEADER]: Becoming Leader. State: Replica: fcd5a6df75864d7ea35630d318368cd3, State: Running, Role: LEADER
I20260619 14:54:27.106063 2561 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5a6df75864d7ea35630d318368cd3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43807 } }
I20260619 14:54:27.106173 2559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.094041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51210 (local address 127.19.34.126:43807)
0619 14:54:27.094418 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:27.094429 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:27.094458 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:54:27.094996 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:27.095009 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:27.095304 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:27.095609 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:27.096357 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.097283 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.098230 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.098702 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.098856 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.101238 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:27.101242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:27.101252 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:27.101273 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:27.103107 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:27.103539 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:27.103543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:27.103547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:27.103643 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:27.103877 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:27.103882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:27.103884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:27.104378 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:27.104573 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:27.104857 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 14:54:27.105220 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260619 14:54:27.106551 2556 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:27.107122 2562 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcd5a6df75864d7ea35630d318368cd3. Latest consensus state: current_term: 1 leader_uuid: "fcd5a6df75864d7ea35630d318368cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5a6df75864d7ea35630d318368cd3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43807 } } }
I20260619 14:54:27.107234 2562 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:27.107853 2563 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcd5a6df75864d7ea35630d318368cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5a6df75864d7ea35630d318368cd3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43807 } } }
I20260619 14:54:27.107951 2563 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:27.108274 2568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:27.110885 2568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:27.115784 2568 catalog_manager.cc:1347] Generated new cluster ID: c55fcd9499784bf28c151330182c5c49
I20260619 14:54:27.115840 2568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:27.131301 2568 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:54:27.132072 2568 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:27.140013 2568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: Generated new TSK 0
I20260619 14:54:27.140631 2568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.34.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
I20260619 14:54:27.170449 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:43807
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:27.283627 2582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:27.283843 2582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:27.283882 2582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:27.287755 2582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:27.287819 2582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:27.287927 2582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:54:27.292594 2582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:43807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:27.293449 2582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:27.294320 2582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:27.302846 2590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:27.302855 2587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:27.302970 2582 server_base.cc:1034] running on GCE node
W20260619 14:54:27.303153 2588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:27.303511 2582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:27.304029 2582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:27.305197 2582 hybrid_clock.cc:648] HybridClock initialized: now 1781880867305163 us; error 46 us; skew 500 ppm
Jun 19 14:54:27 dist-test-slave-3qdh krb5kdc[2475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880867, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:27.308023 2582 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:54:27.308810 2582 webserver.cc:460] Webserver started at http://127.19.34.65:34963/ using document root <none> and password file <none>
I20260619 14:54:27.309279 2582 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:27.309358 2582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:27.309569 2582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:27.311333 2582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "3c5dbac576ad47e1adcf37f7518bc5c7"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-3qdh"
server_key: "b36d70874a1c9597c78c8cf0dc43f62e"
server_key_iv: "371c241bdb57a724340565d4a3e94999"
server_key_version: "encryptionkey@0"
I20260619 14:54:27.311838 2582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "3c5dbac576ad47e1adcf37f7518bc5c7"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-3qdh"
server_key: "b36d70874a1c9597c78c8cf0dc43f62e"
server_key_iv: "371c241bdb57a724340565d4a3e94999"
server_key_version: "encryptionkey@0"
I20260619 14:54:27.315174 2582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:54:27.317328 2597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:27.318413 2582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:54:27.318558 2582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "3c5dbac576ad47e1adcf37f7518bc5c7"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-3qdh"
server_key: "b36d70874a1c9597c78c8cf0dc43f62e"
server_key_iv: "371c241bdb57a724340565d4a3e94999"
server_key_version: "encryptionkey@0"
I20260619 14:54:27.318671 2582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:27.333956 2582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:27.336743 2582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:27.336911 2582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:27.337411 2582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:27.338099 2582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:27.338169 2582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:27.338244 2582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:27.338289 2582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:27.346318 2582 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:34725
I20260619 14:54:27.346352 2710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:34725 every 8 connection(s)
I20260619 14:54:27.347059 2582 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
Jun 19 14:54:27 dist-test-slave-3qdh krb5kdc[2475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880867, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:54:27.355722 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 2582
I20260619 14:54:27.355849 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:54:27.356105 19593 external_mini_cluster.cc:1428] Setting key 99475aad6036bfbdeda6a6daf669dc04
I20260619 14:54:27.359443 2559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.348602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:51293 (local address 127.19.34.126:43807)
0619 14:54:27.348752 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:27.348757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:27.349467 (+ 710us) server_negotiation.cc:407] Connection header received
0619 14:54:27.350617 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:27.350620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:27.350665 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:27.350731 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:27.351792 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.352370 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.353409 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.353635 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.353712 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.355794 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:27.355799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:27.355802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:27.355829 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:27.357438 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:27.357920 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:27.357925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:27.357926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:27.358010 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:27.358336 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:27.358341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:27.358343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:27.358526 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:27.358628 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:27.359237 (+ 609us) server_negotiation.cc:299] Negotiation successful
0619 14:54:27.359332 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:54:27.360445 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.348932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.65:51293)
0619 14:54:27.349326 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:27.349363 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:27.350447 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:27.350840 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:27.350845 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:27.351149 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:27.351633 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:27.351654 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.352479 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.352482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:27.353293 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:27.353301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.353788 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.353791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:27.353911 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.354328 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:27.354342 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:27.355640 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:27.357552 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:27.357557 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:27.357569 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:27.357818 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:27.358100 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:27.358105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:27.358107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:27.358244 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:27.358627 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:27.358633 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:27.358981 (+ 348us) client_negotiation.cc:769] Sending connection context
0619 14:54:27.359186 (+ 205us) client_negotiation.cc:240] Negotiation successful
0619 14:54:27.359495 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":97,"threads_started":1}
I20260619 14:54:27.361501 2711 heartbeater.cc:347] Connected to a master server at 127.19.34.126:43807
I20260619 14:54:27.361737 2711 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:27.362231 2711 heartbeater.cc:511] Master 127.19.34.126:43807 requested a full tablet report, sending...
I20260619 14:54:27.363660 2520 ts_manager.cc:194] Registered new tserver with Master: 3c5dbac576ad47e1adcf37f7518bc5c7 (127.19.34.65:34725)
I20260619 14:54:27.364766 2520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:51293
WARNING: no policy specified for kudu/127.19.34.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
I20260619 14:54:27.381031 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:43807
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:27.476635 2716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:27.476832 2716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:27.476867 2716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:27.480389 2716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:27.480443 2716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:27.480554 2716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:54:27.485208 2716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:43807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:27.485946 2716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:27.486747 2716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:27.495265 2724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:27.495275 2722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:27.495276 2721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:27.495316 2716 server_base.cc:1034] running on GCE node
I20260619 14:54:27.495782 2716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:27.496368 2716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:27.497553 2716 hybrid_clock.cc:648] HybridClock initialized: now 1781880867497524 us; error 49 us; skew 500 ppm
Jun 19 14:54:27 dist-test-slave-3qdh krb5kdc[2475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880867, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:27.500198 2716 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:54:27.501025 2716 webserver.cc:460] Webserver started at http://127.19.34.66:36785/ using document root <none> and password file <none>
I20260619 14:54:27.501489 2716 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:27.501544 2716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:27.501691 2716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:27.503458 2716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "c353bc3ddb754f89a860026be0c983fc"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-3qdh"
server_key: "9c696cfacfd973538cbd0c0c9252494b"
server_key_iv: "8d2c70f7cf838eee77cd60df88fd0e70"
server_key_version: "encryptionkey@0"
I20260619 14:54:27.503882 2716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "c353bc3ddb754f89a860026be0c983fc"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-3qdh"
server_key: "9c696cfacfd973538cbd0c0c9252494b"
server_key_iv: "8d2c70f7cf838eee77cd60df88fd0e70"
server_key_version: "encryptionkey@0"
I20260619 14:54:27.507256 2716 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:54:27.509580 2731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:27.510643 2716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:54:27.510733 2716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "c353bc3ddb754f89a860026be0c983fc"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-3qdh"
server_key: "9c696cfacfd973538cbd0c0c9252494b"
server_key_iv: "8d2c70f7cf838eee77cd60df88fd0e70"
server_key_version: "encryptionkey@0"
I20260619 14:54:27.510807 2716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:27.525830 2716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:27.528718 2716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:27.528848 2716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:27.529310 2716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:27.530020 2716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:27.530063 2716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:27.530105 2716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:27.530126 2716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:27.538259 2716 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:35273
I20260619 14:54:27.538280 2844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:35273 every 8 connection(s)
I20260619 14:54:27.538990 2716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:54:27.545903 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 2716
I20260619 14:54:27.545996 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:54:27.546346 19593 external_mini_cluster.cc:1428] Setting key b64346d0e5f35979a6972626b8786361
Jun 19 14:54:27 dist-test-slave-3qdh krb5kdc[2475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880867, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:54:27.552170 2559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.540594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:49297 (local address 127.19.34.126:43807)
0619 14:54:27.540724 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:27.540728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:27.541818 (+ 1090us) server_negotiation.cc:407] Connection header received
0619 14:54:27.542612 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:27.542615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:27.542658 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:27.542730 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:27.543740 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.544256 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.545325 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.545532 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.545616 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.548220 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:27.548223 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:27.548225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:27.548248 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:27.550194 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:27.550686 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:27.550692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:27.550697 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:27.550791 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:27.551132 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:27.551136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:27.551138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:27.551315 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:27.551420 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:27.551869 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 14:54:27.551979 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:54:27.552922 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.540899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.66:49297)
0619 14:54:27.541678 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:27.541713 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:27.542401 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:27.542862 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:27.542868 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:27.543165 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:27.543601 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:27.543614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.544420 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.544423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:27.545217 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:27.545226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.545672 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.545674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:27.545812 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.546356 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:27.546372 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:27.548042 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:27.550317 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:27.550322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:27.550332 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:27.550566 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:27.550875 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:27.550877 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:27.550879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:27.550999 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:27.551418 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:27.551424 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:27.551645 (+ 221us) client_negotiation.cc:769] Sending connection context
0619 14:54:27.551815 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 14:54:27.552094 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":497,"threads_started":1}
I20260619 14:54:27.554203 2845 heartbeater.cc:347] Connected to a master server at 127.19.34.126:43807
I20260619 14:54:27.554405 2845 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:27.554876 2845 heartbeater.cc:511] Master 127.19.34.126:43807 requested a full tablet report, sending...
I20260619 14:54:27.556000 2520 ts_manager.cc:194] Registered new tserver with Master: c353bc3ddb754f89a860026be0c983fc (127.19.34.66:35273)
WARNING: no policy specified for kudu/127.19.34.67@KRBTEST.COM; defaulting to no policy
I20260619 14:54:27.556892 2520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:49297
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
I20260619 14:54:27.572966 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:43807
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:27.671684 2850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:27.671912 2850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:27.672003 2850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:27.675603 2850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:27.675680 2850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:27.675760 2850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:54:27.680413 2850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:43807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:27.681226 2850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:27.682058 2850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:27.690518 2856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:27.690510 2855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:27.690513 2858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:27.690800 2850 server_base.cc:1034] running on GCE node
I20260619 14:54:27.691192 2850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:27.691803 2850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:27.692961 2850 hybrid_clock.cc:648] HybridClock initialized: now 1781880867692934 us; error 39 us; skew 500 ppm
Jun 19 14:54:27 dist-test-slave-3qdh krb5kdc[2475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880867, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:27.695391 2850 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:54:27.696118 2850 webserver.cc:460] Webserver started at http://127.19.34.67:37131/ using document root <none> and password file <none>
I20260619 14:54:27.696668 2850 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:27.696725 2850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:27.696866 2850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:54:27.698683 2850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "32ad88aff25b4b078b13d2bbc475dd05"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-3qdh"
server_key: "a8c7766854e882e498b2a81052b1edc6"
server_key_iv: "79cde5ab4c6758cc0ccde9cc9395cd7c"
server_key_version: "encryptionkey@0"
I20260619 14:54:27.699124 2850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "32ad88aff25b4b078b13d2bbc475dd05"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-3qdh"
server_key: "a8c7766854e882e498b2a81052b1edc6"
server_key_iv: "79cde5ab4c6758cc0ccde9cc9395cd7c"
server_key_version: "encryptionkey@0"
I20260619 14:54:27.702266 2850 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:54:27.704239 2865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:27.705313 2850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:54:27.705417 2850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "32ad88aff25b4b078b13d2bbc475dd05"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-3qdh"
server_key: "a8c7766854e882e498b2a81052b1edc6"
server_key_iv: "79cde5ab4c6758cc0ccde9cc9395cd7c"
server_key_version: "encryptionkey@0"
I20260619 14:54:27.705515 2850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:27.725584 2850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:27.728004 2850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:27.728195 2850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:27.728740 2850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:27.729394 2850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:27.729435 2850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:27.729468 2850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:27.729487 2850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:27.737303 2850 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:35341
I20260619 14:54:27.737331 2978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:35341 every 8 connection(s)
I20260619 14:54:27.738094 2850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
Jun 19 14:54:27 dist-test-slave-3qdh krb5kdc[2475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880867, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:54:27.748060 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 2850
I20260619 14:54:27.748200 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:54:27.748528 19593 external_mini_cluster.cc:1428] Setting key 82ed5c427ec2a8ceb298823a789bc7ec
I20260619 14:54:27.750326 2559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.739674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:41759 (local address 127.19.34.126:43807)
0619 14:54:27.739820 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:27.739825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:27.740576 (+ 751us) server_negotiation.cc:407] Connection header received
0619 14:54:27.741776 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:27.741779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:27.741822 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:27.741888 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:27.742844 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.743392 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.744402 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.744612 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.744695 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.746851 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:27.746853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:27.746855 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:27.746876 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:27.748076 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:27.748701 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:27.748704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:27.748706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:27.748791 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:27.749205 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:27.749210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:27.749215 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:27.749454 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:27.749539 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:27.750034 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 14:54:27.750171 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:54:27.751012 2981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.739994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.67:41759)
0619 14:54:27.740450 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:27.740487 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:27.741609 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:27.742024 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:27.742030 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:27.742340 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:27.742726 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:27.742737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.743494 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.743496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:27.744297 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:27.744323 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.744751 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.744753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:27.744875 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.745421 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:27.745436 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:27.746714 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:27.748220 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:27.748225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:27.748237 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:27.748568 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:27.748897 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:27.748900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:27.748902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:27.749092 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:27.749553 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:27.749557 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:27.749808 (+ 251us) client_negotiation.cc:769] Sending connection context
0619 14:54:27.749976 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 14:54:27.750264 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":92,"threads_started":1}
I20260619 14:54:27.751895 2979 heartbeater.cc:347] Connected to a master server at 127.19.34.126:43807
I20260619 14:54:27.752092 2979 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:27.752594 2979 heartbeater.cc:511] Master 127.19.34.126:43807 requested a full tablet report, sending...
I20260619 14:54:27.753537 2520 ts_manager.cc:194] Registered new tserver with Master: 32ad88aff25b4b078b13d2bbc475dd05 (127.19.34.67:35341)
I20260619 14:54:27.754076 2520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:41759
I20260619 14:54:27.761896 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:54:27.770689 2559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.763703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51222 (local address 127.19.34.126:43807)
0619 14:54:27.763846 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:27.763850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:27.764013 (+ 163us) server_negotiation.cc:407] Connection header received
0619 14:54:27.764153 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:27.764155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:27.764187 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:27.764287 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:27.764961 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.765436 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.766249 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.766461 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.766565 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.767749 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:27.767753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:27.767755 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:27.767785 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:27.768909 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:27.769328 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:27.769338 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:27.769340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:27.769416 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:27.769862 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:27.769867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:27.769869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:27.770079 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:27.770217 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:27.770436 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 14:54:27.770537 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:54:27.772957 2520 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:54:27.775552 2520 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:54:27.788275 2990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.782772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:35341 (local address 127.0.0.1:58030)
0619 14:54:27.783187 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:27.783227 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:27.783333 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:27.784116 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:27.784122 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:27.784141 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:27.784266 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:27.784272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.785567 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.785570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:27.786788 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:27.786798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.787837 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.787840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:27.787917 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.787947 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 14:54:27.787991 (+ 44us) client_negotiation.cc:240] Negotiation successful
0619 14:54:27.788113 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":36,"thread_start_us":55,"threads_started":1}
I20260619 14:54:27.788285 2993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.783472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45302 (local address 127.19.34.66:35273)
0619 14:54:27.783744 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:27.783755 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:27.784288 (+ 533us) server_negotiation.cc:407] Connection header received
0619 14:54:27.784385 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:27.784391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:27.784548 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:27.784706 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:27.784876 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.785599 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.786961 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.787572 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.787677 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.787817 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:27.788022 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 14:54:27.788119 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":106,"threads_started":1}
I20260619 14:54:27.788511 2991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.782878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58030 (local address 127.19.34.67:35341)
0619 14:54:27.783402 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:27.783413 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:27.783433 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:54:27.783509 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:27.783516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:27.783954 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:27.784430 (+ 476us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:27.784550 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.785447 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.786931 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.787701 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.788100 (+ 399us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.788228 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:27.788275 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 14:54:27.788380 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":126,"threads_started":1}
I20260619 14:54:27.788756 2994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.782937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:35273 (local address 127.0.0.1:45302)
0619 14:54:27.784031 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:27.784045 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:27.784118 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:27.784705 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:27.784708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:27.784724 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:27.784793 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:27.784797 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.785720 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.785723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:27.786847 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:27.786856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.787692 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.787696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:27.787830 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.787873 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 14:54:27.787979 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 14:54:27.788639 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"mutex_wait_us":22,"thread_start_us":69,"threads_started":1}
I20260619 14:54:27.789232 2992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.783606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34725 (local address 127.0.0.1:48886)
0619 14:54:27.784279 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:27.784290 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:27.784390 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:27.784876 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:27.784879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:27.784895 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:27.784970 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:27.784975 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.786166 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.786169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:27.787320 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:27.787329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.788783 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.788786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:27.788957 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.788977 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:54:27.789051 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 14:54:27.789117 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":100,"threads_started":1}
I20260619 14:54:27.790921 2913 tablet_service.cc:1467] Processing CreateTablet for tablet 470263d173c34f839dfd841089586572 (DEFAULT_TABLE table=test-table [id=28583bdb8cc34636baac1f7bb84d06ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:27.790943 2779 tablet_service.cc:1467] Processing CreateTablet for tablet 470263d173c34f839dfd841089586572 (DEFAULT_TABLE table=test-table [id=28583bdb8cc34636baac1f7bb84d06ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:27.791725 2779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470263d173c34f839dfd841089586572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:27.791729 2913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470263d173c34f839dfd841089586572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:27.797132 2997 tablet_bootstrap.cc:492] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc: Bootstrap starting.
I20260619 14:54:27.797555 2996 tablet_bootstrap.cc:492] T 470263d173c34f839dfd841089586572 P 32ad88aff25b4b078b13d2bbc475dd05: Bootstrap starting.
I20260619 14:54:27.789394 2995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.784014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48886 (local address 127.19.34.65:34725)
0619 14:54:27.784369 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:27.784379 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:27.784395 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:54:27.784479 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:27.784482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:27.784610 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:27.784728 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:27.785104 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.786038 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.787445 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.788274 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.788785 (+ 511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.788999 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:27.789096 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 14:54:27.789172 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":109,"threads_started":1}
I20260619 14:54:27.799603 2997 tablet_bootstrap.cc:654] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:27.799712 2996 tablet_bootstrap.cc:654] T 470263d173c34f839dfd841089586572 P 32ad88aff25b4b078b13d2bbc475dd05: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:27.800382 2997 log.cc:824] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:27.800917 2996 log.cc:824] T 470263d173c34f839dfd841089586572 P 32ad88aff25b4b078b13d2bbc475dd05: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:27.800568 2645 tablet_service.cc:1467] Processing CreateTablet for tablet 470263d173c34f839dfd841089586572 (DEFAULT_TABLE table=test-table [id=28583bdb8cc34636baac1f7bb84d06ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:27.801491 2645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 470263d173c34f839dfd841089586572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:27.802090 2997 tablet_bootstrap.cc:492] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc: No bootstrap required, opened a new log
I20260619 14:54:27.802292 2997 ts_tablet_manager.cc:1397] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:54:27.802639 2996 tablet_bootstrap.cc:492] T 470263d173c34f839dfd841089586572 P 32ad88aff25b4b078b13d2bbc475dd05: No bootstrap required, opened a new log
I20260619 14:54:27.802796 2996 ts_tablet_manager.cc:1397] T 470263d173c34f839dfd841089586572 P 32ad88aff25b4b078b13d2bbc475dd05: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:54:27.805217 2997 raft_consensus.cc:348] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:54:27.805426 2997 raft_consensus.cc:374] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:27.805481 2997 raft_consensus.cc:729] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c353bc3ddb754f89a860026be0c983fc, State: Initialized, Role: FOLLOWER
I20260619 14:54:27.805919 2997 consensus_queue.cc:260] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:54:27.806116 2996 raft_consensus.cc:348] T 470263d173c34f839dfd841089586572 P 32ad88aff25b4b078b13d2bbc475dd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:54:27.806360 2996 raft_consensus.cc:374] T 470263d173c34f839dfd841089586572 P 32ad88aff25b4b078b13d2bbc475dd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:27.806417 2996 raft_consensus.cc:729] T 470263d173c34f839dfd841089586572 P 32ad88aff25b4b078b13d2bbc475dd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32ad88aff25b4b078b13d2bbc475dd05, State: Initialized, Role: FOLLOWER
I20260619 14:54:27.806710 2845 heartbeater.cc:503] Master 127.19.34.126:43807 was elected leader, sending a full tablet report...
I20260619 14:54:27.806828 2996 consensus_queue.cc:260] T 470263d173c34f839dfd841089586572 P 32ad88aff25b4b078b13d2bbc475dd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:54:27.806895 2997 ts_tablet_manager.cc:1428] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:54:27.807502 2979 heartbeater.cc:503] Master 127.19.34.126:43807 was elected leader, sending a full tablet report...
I20260619 14:54:27.807700 2996 ts_tablet_manager.cc:1428] T 470263d173c34f839dfd841089586572 P 32ad88aff25b4b078b13d2bbc475dd05: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:54:27.808686 3002 tablet_bootstrap.cc:492] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7: Bootstrap starting.
I20260619 14:54:27.811432 3002 tablet_bootstrap.cc:654] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:27.812156 3002 log.cc:824] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:27.813773 3002 tablet_bootstrap.cc:492] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7: No bootstrap required, opened a new log
I20260619 14:54:27.813934 3002 ts_tablet_manager.cc:1397] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:54:27.816533 3002 raft_consensus.cc:348] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:54:27.816736 3002 raft_consensus.cc:374] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:27.816807 3002 raft_consensus.cc:729] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5dbac576ad47e1adcf37f7518bc5c7, State: Initialized, Role: FOLLOWER
I20260619 14:54:27.817137 3002 consensus_queue.cc:260] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:54:27.817664 2711 heartbeater.cc:503] Master 127.19.34.126:43807 was elected leader, sending a full tablet report...
I20260619 14:54:27.817835 3002 ts_tablet_manager.cc:1428] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260619 14:54:27.844211 3000 raft_consensus.cc:482] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:54:27.844393 3000 raft_consensus.cc:504] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:54:27.845340 3000 leader_election.cc:288] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32ad88aff25b4b078b13d2bbc475dd05 (127.19.34.67:35341), 3c5dbac576ad47e1adcf37f7518bc5c7 (127.19.34.65:34725)
W20260619 14:54:27.847999 2712 tablet.cc:2369] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:54:27.848807 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.845481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:35341 (local address 127.19.34.66:33721)
0619 14:54:27.845642 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:27.845656 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:27.845741 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:27.845971 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:27.845974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:27.845990 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:27.846097 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:27.846102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.846781 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.846787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:27.847545 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:27.847553 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.848390 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.848393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:27.848507 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.848536 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:54:27.848600 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 14:54:27.848696 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":21}
I20260619 14:54:27.848809 2991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.845555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:33721 (local address 127.19.34.67:35341)
0619 14:54:27.845659 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:27.845664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:27.845755 (+ 91us) server_negotiation.cc:407] Connection header received
0619 14:54:27.845808 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:27.845810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:27.845861 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:27.845950 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:27.846204 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.846680 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.847662 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.848240 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.848395 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.848438 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:27.848630 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 14:54:27.848693 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:54:27.848995 2995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.845605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:43993 (local address 127.19.34.65:34725)
0619 14:54:27.845716 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:27.845720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:27.845886 (+ 166us) server_negotiation.cc:407] Connection header received
0619 14:54:27.845984 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:27.845986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:27.846029 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:27.846108 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:27.846374 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.846827 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.847877 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.848461 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.848558 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.848622 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:27.848829 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 14:54:27.848895 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:54:27.849025 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.845536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34725 (local address 127.19.34.66:43993)
0619 14:54:27.845792 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:27.845804 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:27.845904 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:27.846137 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:27.846140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:27.846154 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:27.846265 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:27.846270 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.846934 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.846938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:27.847739 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:27.847747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.848570 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.848592 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:27.848712 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.848732 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:54:27.848800 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 14:54:27.848870 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":21,"thread_start_us":103,"threads_started":1}
I20260619 14:54:27.849510 2933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470263d173c34f839dfd841089586572" candidate_uuid: "c353bc3ddb754f89a860026be0c983fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32ad88aff25b4b078b13d2bbc475dd05" is_pre_election: true
I20260619 14:54:27.849550 2665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470263d173c34f839dfd841089586572" candidate_uuid: "c353bc3ddb754f89a860026be0c983fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" is_pre_election: true
I20260619 14:54:27.849740 2933 raft_consensus.cc:2454] T 470263d173c34f839dfd841089586572 P 32ad88aff25b4b078b13d2bbc475dd05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c353bc3ddb754f89a860026be0c983fc in term 0.
I20260619 14:54:27.849789 2665 raft_consensus.cc:2454] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c353bc3ddb754f89a860026be0c983fc in term 0.
I20260619 14:54:27.850188 2733 leader_election.cc:302] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32ad88aff25b4b078b13d2bbc475dd05, c353bc3ddb754f89a860026be0c983fc; no voters:
I20260619 14:54:27.850394 3000 raft_consensus.cc:2781] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:54:27.850493 3000 raft_consensus.cc:482] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:54:27.850540 3000 raft_consensus.cc:3037] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:27.851449 3000 raft_consensus.cc:504] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:54:27.851752 3000 leader_election.cc:288] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc [CANDIDATE]: Term 1 election: Requested vote from peers 32ad88aff25b4b078b13d2bbc475dd05 (127.19.34.67:35341), 3c5dbac576ad47e1adcf37f7518bc5c7 (127.19.34.65:34725)
I20260619 14:54:27.852044 2933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470263d173c34f839dfd841089586572" candidate_uuid: "c353bc3ddb754f89a860026be0c983fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32ad88aff25b4b078b13d2bbc475dd05"
I20260619 14:54:27.852110 2665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "470263d173c34f839dfd841089586572" candidate_uuid: "c353bc3ddb754f89a860026be0c983fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7"
I20260619 14:54:27.852156 2933 raft_consensus.cc:3037] T 470263d173c34f839dfd841089586572 P 32ad88aff25b4b078b13d2bbc475dd05 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:27.852250 2665 raft_consensus.cc:3037] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:27.853403 2665 raft_consensus.cc:2454] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c353bc3ddb754f89a860026be0c983fc in term 1.
I20260619 14:54:27.853387 2933 raft_consensus.cc:2454] T 470263d173c34f839dfd841089586572 P 32ad88aff25b4b078b13d2bbc475dd05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c353bc3ddb754f89a860026be0c983fc in term 1.
I20260619 14:54:27.853693 2733 leader_election.cc:302] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32ad88aff25b4b078b13d2bbc475dd05, c353bc3ddb754f89a860026be0c983fc; no voters:
I20260619 14:54:27.853917 3000 raft_consensus.cc:2781] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:27.854089 3000 raft_consensus.cc:686] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc [term 1 LEADER]: Becoming Leader. State: Replica: c353bc3ddb754f89a860026be0c983fc, State: Running, Role: LEADER
I20260619 14:54:27.854370 3000 consensus_queue.cc:237] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:54:27.856882 2519 catalog_manager.cc:5495] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc reported cstate change: term changed from 0 to 1, leader changed from <none> to c353bc3ddb754f89a860026be0c983fc (127.19.34.66). New cstate: current_term: 1 leader_uuid: "c353bc3ddb754f89a860026be0c983fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } health_report { overall_health: UNKNOWN } } }
I20260619 14:54:27.882956 2993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.879797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45306 (local address 127.19.34.66:35273)
0619 14:54:27.879915 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:27.879921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:27.879939 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:54:27.880112 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:27.880115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:27.880169 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:27.880266 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:54:27.880507 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.880955 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.881704 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:27.881902 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:27.881996 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:27.882287 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:54:27.882618 (+ 331us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:54:27.882734 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:27.882784 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 14:54:27.882826 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:54:27.887795 2665 raft_consensus.cc:1261] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Refusing update from remote peer c353bc3ddb754f89a860026be0c983fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:27.887799 2933 raft_consensus.cc:1261] T 470263d173c34f839dfd841089586572 P 32ad88aff25b4b078b13d2bbc475dd05 [term 1 FOLLOWER]: Refusing update from remote peer c353bc3ddb754f89a860026be0c983fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:27.888473 3006 consensus_queue.cc:1035] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:27.888661 3000 consensus_queue.cc:1035] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:27.896194 3011 mvcc.cc:205] Tried to move back new op lower bound from 7298584034862157824 to 7298584034733613056. Current Snapshot: MvccSnapshot[applied={T|T < 7298584034862157824}]
I20260619 14:54:27.901914 2520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:27.902047 2520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:27.904119 2520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: Sending DeleteTablet for 3 replicas of tablet 470263d173c34f839dfd841089586572
I20260619 14:54:27.904749 2913 tablet_service.cc:1514] Processing DeleteTablet for tablet 470263d173c34f839dfd841089586572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:27 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:58030
I20260619 14:54:27.904834 2779 tablet_service.cc:1514] Processing DeleteTablet for tablet 470263d173c34f839dfd841089586572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:27 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:45302
I20260619 14:54:27.904910 2645 tablet_service.cc:1514] Processing DeleteTablet for tablet 470263d173c34f839dfd841089586572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:27 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:48886
I20260619 14:54:27.905283 3021 tablet_replica.cc:330] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7: stopping tablet replica
I20260619 14:54:27.905340 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 2582
I20260619 14:54:27.905561 3021 raft_consensus.cc:2229] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:54:27.906076 3020 tablet_replica.cc:330] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc: stopping tablet replica
I20260619 14:54:27.906340 3020 raft_consensus.cc:2229] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:54:27.906629 3020 raft_consensus.cc:2258] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:27.907941 3020 ts_tablet_manager.cc:1905] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:27.910401 3020 ts_tablet_manager.cc:1918] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:27.910463 3020 log.cc:1196] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/wal/wals/470263d173c34f839dfd841089586572
I20260619 14:54:27.910714 3020 ts_tablet_manager.cc:1939] T 470263d173c34f839dfd841089586572 P c353bc3ddb754f89a860026be0c983fc: Deleting consensus metadata
I20260619 14:54:27.911584 2507 catalog_manager.cc:4841] TS c353bc3ddb754f89a860026be0c983fc (127.19.34.66:35273): tablet 470263d173c34f839dfd841089586572 (table test-table [id=28583bdb8cc34636baac1f7bb84d06ea]) successfully deleted
W20260619 14:54:27.912314 2504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.34.65:34725 (error 108)
I20260619 14:54:27.914450 2992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:27.914234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34725 (local address 127.0.0.1:48902)
0619 14:54:27.914339 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:27.914391 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.65:34725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 14:54:27.914577 2504 catalog_manager.cc:4584] TS 3c5dbac576ad47e1adcf37f7518bc5c7 (127.19.34.65:34725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 470263d173c34f839dfd841089586572: Network error: Client connection negotiation failed: client connection to 127.19.34.65:34725: connect: Connection refused (error 111)
I20260619 14:54:27.914726 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 2716
I20260619 14:54:27.914855 3019 tablet_replica.cc:330] T 470263d173c34f839dfd841089586572 P 32ad88aff25b4b078b13d2bbc475dd05: stopping tablet replica
I20260619 14:54:27.915081 3019 raft_consensus.cc:2229] T 470263d173c34f839dfd841089586572 P 32ad88aff25b4b078b13d2bbc475dd05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:54:27.915388 3019 raft_consensus.cc:2258] T 470263d173c34f839dfd841089586572 P 32ad88aff25b4b078b13d2bbc475dd05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:27.916611 3019 ts_tablet_manager.cc:1905] T 470263d173c34f839dfd841089586572 P 32ad88aff25b4b078b13d2bbc475dd05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:27.919852 3019 ts_tablet_manager.cc:1918] T 470263d173c34f839dfd841089586572 P 32ad88aff25b4b078b13d2bbc475dd05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:27.919942 3019 log.cc:1196] T 470263d173c34f839dfd841089586572 P 32ad88aff25b4b078b13d2bbc475dd05: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/470263d173c34f839dfd841089586572
I20260619 14:54:27.920234 3019 ts_tablet_manager.cc:1939] T 470263d173c34f839dfd841089586572 P 32ad88aff25b4b078b13d2bbc475dd05: Deleting consensus metadata
I20260619 14:54:27.921284 2505 catalog_manager.cc:4841] TS 32ad88aff25b4b078b13d2bbc475dd05 (127.19.34.67:35341): tablet 470263d173c34f839dfd841089586572 (table test-table [id=28583bdb8cc34636baac1f7bb84d06ea]) successfully deleted
I20260619 14:54:27.921731 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 2850
I20260619 14:54:27.926430 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 2488
I20260619 14:54:47.932132 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:43807
--webserver_interface=127.19.34.126
--webserver_port=35039
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:43807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:54:48.030867 3023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:48.031087 3023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:48.031128 3023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:48.034544 3023 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 14:54:48.034583 3023 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 14:54:48.034597 3023 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 14:54:48.034608 3023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:54:48.034623 3023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:48.034658 3023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:48.034689 3023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:54:48.034706 3023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:54:48.039225 3023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:43807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:43807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=35039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:48.040107 3023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:48.040953 3023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:48.052922 3030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:48.052951 3029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:48.053192 3023 server_base.cc:1034] running on GCE node
W20260619 14:54:48.053071 3032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:48.053750 3023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:48.054658 3023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:48.055817 3023 hybrid_clock.cc:648] HybridClock initialized: now 1781880888055807 us; error 20 us; skew 500 ppm
Jun 19 14:54:48 dist-test-slave-3qdh krb5kdc[2475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880888, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:48.064793 3023 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:54:48.065552 3023 webserver.cc:460] Webserver started at http://127.19.34.126:35039/ using document root <none> and password file <none>
I20260619 14:54:48.066054 3023 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:48.066170 3023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:48.069654 3023 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:54:48.071544 3039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:48.072721 3023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:54:48.072849 3023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "fcd5a6df75864d7ea35630d318368cd3"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-3qdh"
server_key: "17ed842b98f4c2e99a1d184bb2730606"
server_key_iv: "5b54375eed258c0ef2c2e011a79df3ba"
server_key_version: "encryptionkey@0"
I20260619 14:54:48.073359 3023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:48.085481 3023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:48.088253 3023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:48.088438 3023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:48.094564 3023 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:43807
I20260619 14:54:48.094815 3091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:43807 every 8 connection(s)
I20260619 14:54:48.095468 3023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:54:48.097882 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 3023
I20260619 14:54:48.098305 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:34725
--local_ip_for_outbound_sockets=127.19.34.65
--tserver_master_addrs=127.19.34.126:43807
--webserver_port=34963
--webserver_interface=127.19.34.65
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:54:48.099582 3092 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:54:48.101704 3092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: Bootstrap starting.
I20260619 14:54:48.123358 3092 log.cc:824] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:48.133247 3092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:54:48.133603 3092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: Bootstrap complete.
I20260619 14:54:48.138351 3092 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5a6df75864d7ea35630d318368cd3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43807 } }
I20260619 14:54:48.138844 3092 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd5a6df75864d7ea35630d318368cd3, State: Initialized, Role: FOLLOWER
I20260619 14:54:48.139355 3092 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5a6df75864d7ea35630d318368cd3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43807 } }
I20260619 14:54:48.139496 3092 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:54:48.139569 3092 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:54:48.139662 3092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:54:48.141850 3092 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5a6df75864d7ea35630d318368cd3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43807 } }
I20260619 14:54:48.142206 3092 leader_election.cc:302] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcd5a6df75864d7ea35630d318368cd3; no voters:
I20260619 14:54:48.142496 3092 leader_election.cc:288] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:54:48.142728 3096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:54:48.143087 3096 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 2 LEADER]: Becoming Leader. State: Replica: fcd5a6df75864d7ea35630d318368cd3, State: Running, Role: LEADER
I20260619 14:54:48.143478 3096 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5a6df75864d7ea35630d318368cd3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43807 } }
I20260619 14:54:48.143705 3092 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:54:48.144964 3098 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcd5a6df75864d7ea35630d318368cd3. Latest consensus state: current_term: 2 leader_uuid: "fcd5a6df75864d7ea35630d318368cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5a6df75864d7ea35630d318368cd3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43807 } } }
I20260619 14:54:48.145087 3098 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:48.145447 3097 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fcd5a6df75864d7ea35630d318368cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5a6df75864d7ea35630d318368cd3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43807 } } }
I20260619 14:54:48.145537 3097 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:54:48.148133 3111 catalog_manager.cc:1259] Loaded cluster ID: c55fcd9499784bf28c151330182c5c49
I20260619 14:54:48.148204 3111 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: loading cluster ID for follower catalog manager: success
I20260619 14:54:48.149988 3111 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: acquiring CA information for follower catalog manager: success
I20260619 14:54:48.151080 3111 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:54:48.151204 3106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:54:48.153064 3106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:54:48.153466 3106 catalog_manager.cc:1259] Loaded cluster ID: c55fcd9499784bf28c151330182c5c49
I20260619 14:54:48.153537 3106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:54:48.154074 3106 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:54:48.154510 3106 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: Loaded TSK: 0
I20260619 14:54:48.166230 3106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: Generated new TSK 1
I20260619 14:54:48.166416 3106 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:54:48.222348 3094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:48.222548 3094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:48.222584 3094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:48.226244 3094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:48.226295 3094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:48.226397 3094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:54:48.231199 3094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:34725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=34963
--tserver_master_addrs=127.19.34.126:43807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:48.232039 3094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:48.232889 3094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:48.242238 3120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:48.242254 3119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:48.242389 3122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:48.242738 3094 server_base.cc:1034] running on GCE node
I20260619 14:54:48.243147 3094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:48.243913 3094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:48.245100 3094 hybrid_clock.cc:648] HybridClock initialized: now 1781880888245073 us; error 44 us; skew 500 ppm
Jun 19 14:54:48 dist-test-slave-3qdh krb5kdc[2475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880888, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:48.248106 3094 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:54:48.249087 3094 webserver.cc:460] Webserver started at http://127.19.34.65:34963/ using document root <none> and password file <none>
I20260619 14:54:48.249616 3094 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:48.249686 3094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:48.253175 3094 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 14:54:48.255014 3129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:48.256129 3094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:54:48.256256 3094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "3c5dbac576ad47e1adcf37f7518bc5c7"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-3qdh"
server_key: "b36d70874a1c9597c78c8cf0dc43f62e"
server_key_iv: "371c241bdb57a724340565d4a3e94999"
server_key_version: "encryptionkey@0"
I20260619 14:54:48.256767 3094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:48.282146 3094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:48.284855 3094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:48.285046 3094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:48.285583 3094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:48.286568 3136 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:54:48.290102 3094 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:54:48.290169 3094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260619 14:54:48.290234 3094 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:54:48.291600 3094 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:54:48.291684 3094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 14:54:48.291901 3136 tablet_bootstrap.cc:492] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7: Bootstrap starting.
I20260619 14:54:48.300655 3094 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:34725
I20260619 14:54:48.300774 3243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:34725 every 8 connection(s)
I20260619 14:54:48.301535 3094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:54:48.305054 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 3094
I20260619 14:54:48.305583 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:35273
--local_ip_for_outbound_sockets=127.19.34.66
--tserver_master_addrs=127.19.34.126:43807
--webserver_port=36785
--webserver_interface=127.19.34.66
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:54:48.309463 3136 log.cc:824] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:54:48 dist-test-slave-3qdh krb5kdc[2475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880888, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:54:48.316880 3136 tablet_bootstrap.cc:492] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:54:48.317292 3136 tablet_bootstrap.cc:492] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7: Bootstrap complete.
I20260619 14:54:48.318490 3136 ts_tablet_manager.cc:1397] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.007s
I20260619 14:54:48.320118 3247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:48.303282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.65:45783)
0619 14:54:48.303823 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:48.303884 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:48.305098 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:48.306406 (+ 1308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:48.306414 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:48.306894 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:48.307720 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:48.307740 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.309221 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.309224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:48.310335 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:48.310345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.311334 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.311342 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:48.311599 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:48.312397 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:48.312422 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:48.314253 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:48.316876 (+ 2623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:48.316881 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:48.316893 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:48.317222 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:48.317643 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:48.317649 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:48.317654 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:48.317818 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:48.318462 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:48.318468 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:48.318741 (+ 273us) client_negotiation.cc:769] Sending connection context
0619 14:54:48.318950 (+ 209us) client_negotiation.cc:240] Negotiation successful
0619 14:54:48.319245 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":140,"threads_started":1}
I20260619 14:54:48.320463 3246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:48.303316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:45783 (local address 127.19.34.126:43807)
0619 14:54:48.303669 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:48.303686 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:48.303997 (+ 311us) server_negotiation.cc:407] Connection header received
0619 14:54:48.305568 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:48.305582 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:48.305978 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:48.306312 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:48.307911 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.309060 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.310504 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.311174 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.311343 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:48.314473 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:48.314480 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:48.314497 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:48.314534 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:48.316752 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:48.317343 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:48.317352 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:48.317360 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:48.317513 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:48.317924 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:48.317928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:48.317929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:48.318343 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:48.318467 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:48.319079 (+ 612us) server_negotiation.cc:299] Negotiation successful
0619 14:54:48.319366 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":102,"threads_started":1}
I20260619 14:54:48.321254 3136 raft_consensus.cc:348] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:54:48.321713 3136 raft_consensus.cc:729] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5dbac576ad47e1adcf37f7518bc5c7, State: Initialized, Role: FOLLOWER
I20260619 14:54:48.322163 3136 consensus_queue.cc:260] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:54:48.322835 3244 heartbeater.cc:347] Connected to a master server at 127.19.34.126:43807
I20260619 14:54:48.322990 3136 ts_tablet_manager.cc:1428] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260619 14:54:48.323030 3244 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:48.323459 3244 heartbeater.cc:511] Master 127.19.34.126:43807 requested a full tablet report, sending...
I20260619 14:54:48.325721 3056 ts_manager.cc:194] Registered new tserver with Master: 3c5dbac576ad47e1adcf37f7518bc5c7 (127.19.34.65:34725)
I20260619 14:54:48.328617 3056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:45783
I20260619 14:54:48.330968 3244 heartbeater.cc:503] Master 127.19.34.126:43807 was elected leader, sending a full tablet report...
Jun 19 14:54:48 dist-test-slave-3qdh krb5kdc[2475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880888, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for kudu/127.19.34.65@KRBTEST.COM
I20260619 14:54:48.337477 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:48.328048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34725 (local address 127.0.0.1:51162)
0619 14:54:48.328704 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:48.328742 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:48.328930 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:48.329445 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:48.329451 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:48.329775 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:48.329864 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:48.329870 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.330762 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.330766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:48.331541 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:48.331550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.332202 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.332206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:48.332357 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:48.332820 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:48.332830 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:48.334153 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:48.336067 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:48.336074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:48.336077 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:48.336290 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:48.336613 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:48.336615 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:48.336616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:48.336689 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:48.337108 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:48.337111 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:48.337186 (+ 75us) client_negotiation.cc:769] Sending connection context
0619 14:54:48.337268 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 14:54:48.337378 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":101,"threads_started":1}
I20260619 14:54:48.337625 3252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:48.328129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51162 (local address 127.19.34.65:34725)
0619 14:54:48.328455 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:48.328465 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:48.328841 (+ 376us) server_negotiation.cc:407] Connection header received
0619 14:54:48.329090 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:48.329095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:48.329284 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:48.329407 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:48.329996 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.330657 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.331651 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.332068 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.332174 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:48.334281 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:48.334290 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:48.334300 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:48.334323 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:48.335962 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:48.336439 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:48.336444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:48.336446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:48.336528 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:48.336784 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:48.336787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:48.336788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:48.337014 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:48.337134 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:48.337370 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 14:54:48.337503 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":89,"threads_started":1}
I20260619 14:54:48.338357 3178 tablet_service.cc:1514] Processing DeleteTablet for tablet 470263d173c34f839dfd841089586572 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:27 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:51162
I20260619 14:54:48.339200 3253 tablet_replica.cc:330] stopping tablet replica
I20260619 14:54:48.339452 3253 raft_consensus.cc:2229] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:54:48.339690 3253 raft_consensus.cc:2258] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:48.340600 3253 ts_tablet_manager.cc:1905] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:48.343786 3253 ts_tablet_manager.cc:1918] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:48.343852 3253 log.cc:1196] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/wal/wals/470263d173c34f839dfd841089586572
I20260619 14:54:48.344188 3253 ts_tablet_manager.cc:1939] T 470263d173c34f839dfd841089586572 P 3c5dbac576ad47e1adcf37f7518bc5c7: Deleting consensus metadata
I20260619 14:54:48.345680 3040 catalog_manager.cc:4841] TS 3c5dbac576ad47e1adcf37f7518bc5c7 (127.19.34.65:34725): tablet 470263d173c34f839dfd841089586572 (table test-table [id=28583bdb8cc34636baac1f7bb84d06ea]) successfully deleted
W20260619 14:54:48.413952 3248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:48.414186 3248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:48.414249 3248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:48.417620 3248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:48.417695 3248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:48.417788 3248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:54:48.422324 3248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:35273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=36785
--tserver_master_addrs=127.19.34.126:43807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:48.423166 3248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:48.424003 3248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:48.432576 3261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:48.432585 3259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:48.432626 3248 server_base.cc:1034] running on GCE node
W20260619 14:54:48.432585 3258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:48.433135 3248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:48.433579 3248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:48.434738 3248 hybrid_clock.cc:648] HybridClock initialized: now 1781880888434721 us; error 44 us; skew 500 ppm
Jun 19 14:54:48 dist-test-slave-3qdh krb5kdc[2475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880888, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:48.437425 3248 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:54:48.438287 3248 webserver.cc:460] Webserver started at http://127.19.34.66:36785/ using document root <none> and password file <none>
I20260619 14:54:48.438756 3248 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:48.438827 3248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:48.442031 3248 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:54:48.443897 3268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:48.445220 3248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:54:48.445324 3248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "c353bc3ddb754f89a860026be0c983fc"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-3qdh"
server_key: "9c696cfacfd973538cbd0c0c9252494b"
server_key_iv: "8d2c70f7cf838eee77cd60df88fd0e70"
server_key_version: "encryptionkey@0"
I20260619 14:54:48.445832 3248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:48.456132 3248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:48.458567 3248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:48.458748 3248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:48.459295 3248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:48.460119 3248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:48.460165 3248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:48.460227 3248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:48.460270 3248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:48.468230 3248 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:35273
I20260619 14:54:48.468286 3381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:35273 every 8 connection(s)
I20260619 14:54:48.469118 3248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:54:48.472723 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 3248
I20260619 14:54:48.473196 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:35341
--local_ip_for_outbound_sockets=127.19.34.67
--tserver_master_addrs=127.19.34.126:43807
--webserver_port=37131
--webserver_interface=127.19.34.67
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:54:48 dist-test-slave-3qdh krb5kdc[2475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880888, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:54:48.481869 3246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:48.470663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:35237 (local address 127.19.34.126:43807)
0619 14:54:48.470814 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:48.470819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:48.471496 (+ 677us) server_negotiation.cc:407] Connection header received
0619 14:54:48.472675 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:48.472679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:48.472733 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:48.472845 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:48.474288 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.474758 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.475742 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.475983 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.476079 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:48.478424 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:48.478427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:48.478429 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:48.478449 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:48.479939 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:48.480512 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:48.480516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:48.480518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:48.480592 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:48.480904 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:48.480908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:48.480910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:48.481060 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:48.481165 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:48.481592 (+ 427us) server_negotiation.cc:299] Negotiation successful
0619 14:54:48.481711 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:54:48.482542 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:48.470975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.66:35237)
0619 14:54:48.471367 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:48.471401 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:48.472483 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:48.473060 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:48.473069 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:48.473522 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:48.474143 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:48.474163 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.474878 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.474881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:48.475626 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:48.475634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.476152 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.476156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:48.476358 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:48.476912 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:48.476931 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:48.478272 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:48.480100 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:48.480107 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:48.480119 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:48.480420 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:48.480678 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:48.480681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:48.480683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:48.480806 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:48.481163 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:48.481167 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:48.481369 (+ 202us) client_negotiation.cc:769] Sending connection context
0619 14:54:48.481522 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 14:54:48.481814 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":90,"threads_started":1}
I20260619 14:54:48.483556 3382 heartbeater.cc:347] Connected to a master server at 127.19.34.126:43807
I20260619 14:54:48.483759 3382 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:48.484157 3382 heartbeater.cc:511] Master 127.19.34.126:43807 requested a full tablet report, sending...
I20260619 14:54:48.485052 3056 ts_manager.cc:194] Registered new tserver with Master: c353bc3ddb754f89a860026be0c983fc (127.19.34.66:35273)
I20260619 14:54:48.485529 3056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:35237
W20260619 14:54:48.570914 3385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:54:48.571173 3385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:54:48.571264 3385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:54:48.575206 3385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:54:48.575274 3385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:54:48.575347 3385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:54:48.580422 3385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:35341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=37131
--tserver_master_addrs=127.19.34.126:43807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:54:48.581239 3385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:54:48.582038 3385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:54:48.590377 3391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:48.590380 3390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:54:48.590710 3393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:54:48.590803 3385 server_base.cc:1034] running on GCE node
I20260619 14:54:48.591172 3385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:54:48.591670 3385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:54:48.592809 3385 hybrid_clock.cc:648] HybridClock initialized: now 1781880888592787 us; error 33 us; skew 500 ppm
Jun 19 14:54:48 dist-test-slave-3qdh krb5kdc[2475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880888, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:54:48.595084 3385 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:54:48.595826 3385 webserver.cc:460] Webserver started at http://127.19.34.67:37131/ using document root <none> and password file <none>
I20260619 14:54:48.596345 3385 fs_manager.cc:359] Metadata directory not provided
I20260619 14:54:48.596432 3385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:54:48.600239 3385 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:54:48.601919 3400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:48.602955 3385 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:54:48.603060 3385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "32ad88aff25b4b078b13d2bbc475dd05"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-3qdh"
server_key: "a8c7766854e882e498b2a81052b1edc6"
server_key_iv: "79cde5ab4c6758cc0ccde9cc9395cd7c"
server_key_version: "encryptionkey@0"
I20260619 14:54:48.603489 3385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:54:48.619192 3385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:54:48.621609 3385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:54:48.621773 3385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:54:48.622231 3385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:54:48.622910 3385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:54:48.622973 3385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:48.623031 3385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:54:48.623075 3385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:54:48.635335 3385 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:35341
I20260619 14:54:48.636591 3385 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:54:48.637787 3513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:35341 every 8 connection(s)
I20260619 14:54:48.638919 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 3385
Jun 19 14:54:48 dist-test-slave-3qdh krb5kdc[2475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880888, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:54:48.647734 3517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:48.639722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51112 (local address 127.19.34.126:43807)
0619 14:54:48.639992 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:48.639995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:48.640034 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:54:48.640247 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:48.640251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:48.640329 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:48.640434 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:48.641196 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.641970 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.642767 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.642968 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.643038 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:48.644335 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:48.644340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:48.644342 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:48.644370 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:48.646166 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:48.646551 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:48.646555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:48.646557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:48.646644 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:48.646881 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:48.646884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:48.646886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:48.647126 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:48.647325 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:48.647426 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 14:54:48.647549 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":97,"threads_started":1}
I20260619 14:54:48.650918 3246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:48.639476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:38013 (local address 127.19.34.126:43807)
0619 14:54:48.639624 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:48.639629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:48.640500 (+ 871us) server_negotiation.cc:407] Connection header received
0619 14:54:48.641455 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:48.641458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:48.641492 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:48.641543 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:48.642656 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.643197 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.644573 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.644872 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.644947 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:48.647654 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:48.647657 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:48.647658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:48.647676 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:48.649083 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:48.649802 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:48.649805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:48.649806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:48.649873 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:48.650180 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:48.650182 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:48.650183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:48.650312 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:48.650376 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:48.650764 (+ 388us) server_negotiation.cc:299] Negotiation successful
0619 14:54:48.650834 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:54:48.651652 3518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:48.639855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.67:38013)
0619 14:54:48.640369 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:48.640424 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:48.641258 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:48.641693 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:48.641700 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:48.642130 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:54:48.642526 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:48.642537 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.643367 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.643372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:48.644446 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:48.644456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.645018 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.645025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:48.645260 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:48.645690 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:54:48.645711 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:54:48.647456 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:54:48.649227 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:48.649233 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:48.649246 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:48.649694 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:48.649955 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:54:48.649957 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:54:48.649959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:54:48.650099 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:54:48.650389 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:54:48.650393 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:54:48.650594 (+ 201us) client_negotiation.cc:769] Sending connection context
0619 14:54:48.650728 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 14:54:48.650920 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":140,"threads_started":1}
I20260619 14:54:48.652556 3514 heartbeater.cc:347] Connected to a master server at 127.19.34.126:43807
I20260619 14:54:48.652767 3514 heartbeater.cc:464] Registering TS with master...
I20260619 14:54:48.653193 3514 heartbeater.cc:511] Master 127.19.34.126:43807 requested a full tablet report, sending...
I20260619 14:54:48.654093 3056 ts_manager.cc:194] Registered new tserver with Master: 32ad88aff25b4b078b13d2bbc475dd05 (127.19.34.67:35341)
I20260619 14:54:48.654546 3056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:38013
I20260619 14:54:48.661079 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:54:48.664166 3246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:48.661574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51126 (local address 127.19.34.126:43807)
0619 14:54:48.661691 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:48.661694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:48.661838 (+ 144us) server_negotiation.cc:407] Connection header received
0619 14:54:48.661939 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:48.661941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:48.662015 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:48.662086 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:54:48.662258 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.662704 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.663411 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.663575 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.663671 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:48.663854 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:54:48.663944 (+ 90us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 14:54:48.664091 (+ 147us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51126: token expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 14:54:48.664235 3246 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51126: token expired
W20260619 14:54:48.664573 2984 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.19.34.126:43807)
I20260619 14:54:48.670303 3246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:48.664821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51140 (local address 127.19.34.126:43807)
0619 14:54:48.664895 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:48.664898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:48.664930 (+ 32us) server_negotiation.cc:407] Connection header received
0619 14:54:48.665000 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:48.665002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:48.665027 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:48.665077 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:54:48.665601 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.665982 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.666665 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.666828 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.666891 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:48.667985 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:54:48.667987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:54:48.667989 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:54:48.668005 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:54:48.669150 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:48.669496 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:48.669498 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:48.669499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:48.669561 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:54:48.669773 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:54:48.669775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:54:48.669776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:54:48.669909 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:54:48.670005 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:48.670160 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 14:54:48.670225 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260619 14:54:48.672106 3056 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:54:48.675087 3056 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:54:48.681885 3178 tablet_service.cc:1467] Processing CreateTablet for tablet ed43406a866948c7bc29e3d438d9cd8d (DEFAULT_TABLE table=test-table [id=6ce0f33e830d40a99548de9103026bbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:48.682431 3178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed43406a866948c7bc29e3d438d9cd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:48.684981 3136 tablet_bootstrap.cc:492] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7: Bootstrap starting.
I20260619 14:54:48.685709 3136 tablet_bootstrap.cc:654] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:48.686825 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:48.680979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:35341 (local address 127.0.0.1:45496)
0619 14:54:48.681123 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:48.681135 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:48.681201 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:48.683562 (+ 2361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:48.683564 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:48.683577 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:48.683654 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:48.683660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.684511 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.684514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:48.685524 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:48.685533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.686455 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.686458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:48.686538 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:48.686571 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 14:54:48.686651 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 14:54:48.686721 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":14}
I20260619 14:54:48.686862 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:48.681359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.19.34.67:35341)
0619 14:54:48.683196 (+ 1837us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:48.683204 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:48.683220 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:54:48.683296 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:48.683300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:48.683463 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:48.683586 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:48.683723 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.684405 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.685635 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.686350 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.686456 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:48.686575 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:48.686675 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 14:54:48.686748 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1748,"thread_start_us":103,"threads_started":1}
I20260619 14:54:48.686968 3136 tablet_bootstrap.cc:492] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7: No bootstrap required, opened a new log
I20260619 14:54:48.687039 3136 ts_tablet_manager.cc:1397] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:54:48.687412 3136 raft_consensus.cc:348] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:54:48.687525 3136 raft_consensus.cc:374] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:48.687561 3136 raft_consensus.cc:729] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5dbac576ad47e1adcf37f7518bc5c7, State: Initialized, Role: FOLLOWER
I20260619 14:54:48.687677 3136 consensus_queue.cc:260] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:54:48.687906 3136 ts_tablet_manager.cc:1428] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:54:48.689687 3436 tablet_service.cc:1467] Processing CreateTablet for tablet ed43406a866948c7bc29e3d438d9cd8d (DEFAULT_TABLE table=test-table [id=6ce0f33e830d40a99548de9103026bbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:48.690451 3436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed43406a866948c7bc29e3d438d9cd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:48.695761 3527 tablet_bootstrap.cc:492] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05: Bootstrap starting.
I20260619 14:54:48.697053 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:48.681393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:35273 (local address 127.0.0.1:50368)
0619 14:54:48.682794 (+ 1401us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:48.682805 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:48.682885 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:48.683304 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:48.683307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:48.683323 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:48.683402 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:48.683406 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.684511 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.684514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:48.695768 (+ 11254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:48.695775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.696710 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.696713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:48.696797 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:48.696818 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:54:48.696888 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 14:54:48.696947 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1345,"mutex_wait_us":2,"thread_start_us":78,"threads_started":1}
I20260619 14:54:48.697072 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:48.681762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50368 (local address 127.19.34.66:35273)
0619 14:54:48.682098 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:48.682109 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:48.682873 (+ 764us) server_negotiation.cc:407] Connection header received
0619 14:54:48.682981 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:48.682987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:48.683170 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:48.683309 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:48.683485 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.684429 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.695924 (+ 11495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:48.696600 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:48.696678 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:48.696761 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:48.696913 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 14:54:48.696972 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":109,"threads_started":1}
I20260619 14:54:48.697933 3527 tablet_bootstrap.cc:654] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:48.698661 3527 log.cc:824] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:48.699499 3316 tablet_service.cc:1467] Processing CreateTablet for tablet ed43406a866948c7bc29e3d438d9cd8d (DEFAULT_TABLE table=test-table [id=6ce0f33e830d40a99548de9103026bbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:54:48.700459 3316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed43406a866948c7bc29e3d438d9cd8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:54:48.700552 3527 tablet_bootstrap.cc:492] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05: No bootstrap required, opened a new log
I20260619 14:54:48.700697 3527 ts_tablet_manager.cc:1397] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:54:48.703855 3527 raft_consensus.cc:348] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:54:48.704101 3527 raft_consensus.cc:374] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:48.704162 3527 raft_consensus.cc:729] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32ad88aff25b4b078b13d2bbc475dd05, State: Initialized, Role: FOLLOWER
I20260619 14:54:48.704572 3527 consensus_queue.cc:260] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:54:48.705116 3514 heartbeater.cc:503] Master 127.19.34.126:43807 was elected leader, sending a full tablet report...
I20260619 14:54:48.705433 3527 ts_tablet_manager.cc:1428] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:54:48.707185 3530 tablet_bootstrap.cc:492] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc: Bootstrap starting.
I20260619 14:54:48.709365 3530 tablet_bootstrap.cc:654] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc: Neither blocks nor log segments found. Creating new log.
I20260619 14:54:48.710062 3530 log.cc:824] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:54:48.711534 3530 tablet_bootstrap.cc:492] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc: No bootstrap required, opened a new log
I20260619 14:54:48.711715 3530 ts_tablet_manager.cc:1397] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:54:48.714342 3530 raft_consensus.cc:348] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:54:48.714531 3530 raft_consensus.cc:374] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:54:48.714614 3530 raft_consensus.cc:729] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c353bc3ddb754f89a860026be0c983fc, State: Initialized, Role: FOLLOWER
I20260619 14:54:48.715009 3530 consensus_queue.cc:260] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:54:48.715555 3382 heartbeater.cc:503] Master 127.19.34.126:43807 was elected leader, sending a full tablet report...
I20260619 14:54:48.715809 3530 ts_tablet_manager.cc:1428] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260619 14:54:48.719937 3383 tablet.cc:2369] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:54:48.802736 3245 tablet.cc:2369] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:54:48.888430 3515 tablet.cc:2369] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:54:49.047035 3532 raft_consensus.cc:482] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:54:49.047252 3532 raft_consensus.cc:504] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:54:49.048488 3532 leader_election.cc:288] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32ad88aff25b4b078b13d2bbc475dd05 (127.19.34.67:35341), 3c5dbac576ad47e1adcf37f7518bc5c7 (127.19.34.65:34725)
I20260619 14:54:49.052588 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:49.048865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:38117 (local address 127.19.34.67:35341)
0619 14:54:49.049006 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:49.049010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:49.049174 (+ 164us) server_negotiation.cc:407] Connection header received
0619 14:54:49.049348 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:49.049354 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:49.049400 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:49.049466 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:49.049783 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:49.050494 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:49.051619 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:49.051999 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:49.052085 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:49.052157 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:49.052380 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 14:54:49.052461 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:54:49.052613 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:49.048716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:35341 (local address 127.19.34.66:38117)
0619 14:54:49.049078 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:49.049094 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:49.049224 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:49.049504 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:49.049509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:49.049531 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:49.049666 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:49.049673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:49.050598 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:49.050602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:49.051494 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:49.051502 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:49.052121 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:49.052127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:49.052224 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:49.052246 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:54:49.052360 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 14:54:49.052436 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":114,"thread_start_us":74,"threads_started":1}
I20260619 14:54:49.052640 3535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:49.048827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:38751 (local address 127.19.34.65:34725)
0619 14:54:49.049097 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:49.049102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:49.049181 (+ 79us) server_negotiation.cc:407] Connection header received
0619 14:54:49.049389 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:49.049392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:49.049467 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:49.049575 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:54:49.049873 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:49.050501 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:49.051612 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:49.052035 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:49.052173 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:49.052342 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:49.052441 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 14:54:49.052518 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":86,"threads_started":1}
I20260619 14:54:49.052626 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:49.048716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34725 (local address 127.19.34.66:38751)
0619 14:54:49.049051 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:49.049069 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:54:49.049253 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:54:49.049602 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:54:49.049605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:54:49.049620 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:54:49.049776 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:49.049785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:49.050623 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:49.050626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:49.051492 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:54:49.051500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:54:49.052185 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:54:49.052189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:54:49.052277 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:49.052337 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 14:54:49.052427 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 14:54:49.052487 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":21,"thread_start_us":84,"threads_started":1}
I20260619 14:54:49.053206 3468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed43406a866948c7bc29e3d438d9cd8d" candidate_uuid: "c353bc3ddb754f89a860026be0c983fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32ad88aff25b4b078b13d2bbc475dd05" is_pre_election: true
I20260619 14:54:49.053493 3468 raft_consensus.cc:2454] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c353bc3ddb754f89a860026be0c983fc in term 0.
I20260619 14:54:49.053648 3198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed43406a866948c7bc29e3d438d9cd8d" candidate_uuid: "c353bc3ddb754f89a860026be0c983fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" is_pre_election: true
I20260619 14:54:49.053925 3198 raft_consensus.cc:2454] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c353bc3ddb754f89a860026be0c983fc in term 0.
I20260619 14:54:49.053956 3270 leader_election.cc:302] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32ad88aff25b4b078b13d2bbc475dd05, c353bc3ddb754f89a860026be0c983fc; no voters:
I20260619 14:54:49.054226 3532 raft_consensus.cc:2781] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:54:49.054291 3532 raft_consensus.cc:482] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:54:49.054320 3532 raft_consensus.cc:3037] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:49.055308 3532 raft_consensus.cc:504] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:54:49.055668 3532 leader_election.cc:288] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc [CANDIDATE]: Term 1 election: Requested vote from peers 32ad88aff25b4b078b13d2bbc475dd05 (127.19.34.67:35341), 3c5dbac576ad47e1adcf37f7518bc5c7 (127.19.34.65:34725)
I20260619 14:54:49.055997 3468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed43406a866948c7bc29e3d438d9cd8d" candidate_uuid: "c353bc3ddb754f89a860026be0c983fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32ad88aff25b4b078b13d2bbc475dd05"
I20260619 14:54:49.056017 3198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed43406a866948c7bc29e3d438d9cd8d" candidate_uuid: "c353bc3ddb754f89a860026be0c983fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7"
I20260619 14:54:49.056088 3468 raft_consensus.cc:3037] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:49.056146 3198 raft_consensus.cc:3037] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:54:49.057113 3468 raft_consensus.cc:2454] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c353bc3ddb754f89a860026be0c983fc in term 1.
I20260619 14:54:49.057137 3198 raft_consensus.cc:2454] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c353bc3ddb754f89a860026be0c983fc in term 1.
I20260619 14:54:49.057385 3270 leader_election.cc:302] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32ad88aff25b4b078b13d2bbc475dd05, c353bc3ddb754f89a860026be0c983fc; no voters:
I20260619 14:54:49.057562 3532 raft_consensus.cc:2781] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:54:49.057750 3532 raft_consensus.cc:686] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc [term 1 LEADER]: Becoming Leader. State: Replica: c353bc3ddb754f89a860026be0c983fc, State: Running, Role: LEADER
I20260619 14:54:49.058033 3532 consensus_queue.cc:237] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:54:49.061189 3056 catalog_manager.cc:5495] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc reported cstate change: term changed from 0 to 1, leader changed from <none> to c353bc3ddb754f89a860026be0c983fc (127.19.34.66). New cstate: current_term: 1 leader_uuid: "c353bc3ddb754f89a860026be0c983fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } health_report { overall_health: UNKNOWN } } }
I20260619 14:54:49.134778 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:49.131698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.19.34.66:35273)
0619 14:54:49.131820 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:54:49.131824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:54:49.131858 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:54:49.131925 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:54:49.131927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:54:49.132010 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:54:49.132089 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:54:49.132329 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:49.132804 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:49.133576 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:54:49.133808 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:54:49.133883 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:54:49.134129 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:54:49.134488 (+ 359us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:54:49.134571 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:54:49.134630 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 14:54:49.134677 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:54:49.140122 3468 raft_consensus.cc:1261] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05 [term 1 FOLLOWER]: Refusing update from remote peer c353bc3ddb754f89a860026be0c983fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:49.140170 3198 raft_consensus.cc:1261] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Refusing update from remote peer c353bc3ddb754f89a860026be0c983fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:54:49.140791 3536 consensus_queue.cc:1035] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:49.140971 3532 consensus_queue.cc:1035] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:54:49.146090 3538 mvcc.cc:205] Tried to move back new op lower bound from 7298584121911230464 to 7298584121583964160. Current Snapshot: MvccSnapshot[applied={T|T < 7298584121911230464}]
I20260619 14:54:49.146740 3539 mvcc.cc:205] Tried to move back new op lower bound from 7298584121911230464 to 7298584121583964160. Current Snapshot: MvccSnapshot[applied={T|T < 7298584121911230464}]
I20260619 14:54:49.155236 3056 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:49.155365 3056 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:54:49.157373 3056 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: Sending DeleteTablet for 3 replicas of tablet ed43406a866948c7bc29e3d438d9cd8d
I20260619 14:54:49.157877 3436 tablet_service.cc:1514] Processing DeleteTablet for tablet ed43406a866948c7bc29e3d438d9cd8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:49 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:45496
I20260619 14:54:49.157920 3316 tablet_service.cc:1514] Processing DeleteTablet for tablet ed43406a866948c7bc29e3d438d9cd8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:49 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:50368
I20260619 14:54:49.158057 3178 tablet_service.cc:1514] Processing DeleteTablet for tablet ed43406a866948c7bc29e3d438d9cd8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:54:49 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:51162
I20260619 14:54:49.158273 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 3094
I20260619 14:54:49.158396 3548 tablet_replica.cc:330] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05: stopping tablet replica
I20260619 14:54:49.158661 3548 raft_consensus.cc:2229] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:54:49.158946 3548 raft_consensus.cc:2258] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:49.160246 3548 ts_tablet_manager.cc:1905] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:49.160564 3549 tablet_replica.cc:330] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc: stopping tablet replica
I20260619 14:54:49.160830 3549 raft_consensus.cc:2229] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:54:49.161201 3549 raft_consensus.cc:2258] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:54:49.162578 3549 ts_tablet_manager.cc:1905] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:54:49.162879 3548 ts_tablet_manager.cc:1918] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:49.162998 3548 log.cc:1196] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/ed43406a866948c7bc29e3d438d9cd8d
I20260619 14:54:49.163343 3548 ts_tablet_manager.cc:1939] T ed43406a866948c7bc29e3d438d9cd8d P 32ad88aff25b4b078b13d2bbc475dd05: Deleting consensus metadata
I20260619 14:54:49.164387 3041 catalog_manager.cc:4841] TS 32ad88aff25b4b078b13d2bbc475dd05 (127.19.34.67:35341): tablet ed43406a866948c7bc29e3d438d9cd8d (table test-table [id=6ce0f33e830d40a99548de9103026bbe]) successfully deleted
W20260619 14:54:49.165670 3040 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.34.65:34725 (error 108)
I20260619 14:54:49.166060 3549 ts_tablet_manager.cc:1918] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:54:49.166193 3549 log.cc:1196] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/wal/wals/ed43406a866948c7bc29e3d438d9cd8d
I20260619 14:54:49.166193 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 3248
I20260619 14:54:49.167294 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:49.166975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34725 (local address 127.0.0.1:51174)
0619 14:54:49.167129 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:49.167204 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.65:34725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 14:54:49.167447 3040 catalog_manager.cc:4584] TS 3c5dbac576ad47e1adcf37f7518bc5c7 (127.19.34.65:34725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed43406a866948c7bc29e3d438d9cd8d: Network error: Client connection negotiation failed: client connection to 127.19.34.65:34725: connect: Connection refused (error 111)
I20260619 14:54:49.171610 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 3385
I20260619 14:54:49.171984 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:54:49.171731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:35273 (local address 127.0.0.1:50386)
0619 14:54:49.171868 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 14:54:49.171915 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.66:35273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 14:54:49.176569 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 3023
I20260619 14:55:09.182891 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:43807
--webserver_interface=127.19.34.126
--webserver_port=35039
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:43807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:09.280435 3552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:09.280684 3552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:09.280725 3552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:09.284325 3552 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 14:55:09.284421 3552 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 14:55:09.284453 3552 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 14:55:09.284502 3552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:09.284544 3552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:09.284595 3552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:09.284633 3552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:09.284669 3552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:09.289299 3552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:43807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:43807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=35039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:09.290073 3552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:09.290896 3552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:55:09.298305 3552 server_base.cc:1034] running on GCE node
W20260619 14:55:09.298321 3557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:09.298322 3558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:09.298481 3560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:09.298995 3552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:09.299919 3552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:09.301172 3552 hybrid_clock.cc:648] HybridClock initialized: now 1781880909301136 us; error 32 us; skew 500 ppm
Jun 19 14:55:09 dist-test-slave-3qdh krb5kdc[2475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880909, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:09.307251 3552 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:55:09.307979 3552 webserver.cc:460] Webserver started at http://127.19.34.126:35039/ using document root <none> and password file <none>
I20260619 14:55:09.308444 3552 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:09.308504 3552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:09.311538 3552 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:09.313330 3567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:09.314302 3552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:55:09.314388 3552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "fcd5a6df75864d7ea35630d318368cd3"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-3qdh"
server_key: "17ed842b98f4c2e99a1d184bb2730606"
server_key_iv: "5b54375eed258c0ef2c2e011a79df3ba"
server_key_version: "encryptionkey@0"
I20260619 14:55:09.314780 3552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:09.323668 3552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:09.325851 3552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:09.325963 3552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:09.331903 3552 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:43807
I20260619 14:55:09.332165 3619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:43807 every 8 connection(s)
I20260619 14:55:09.332784 3552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:09.336202 3620 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:55:09.337594 3620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: Bootstrap starting.
I20260619 14:55:09.338337 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 3552
I20260619 14:55:09.338714 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:34725
--local_ip_for_outbound_sockets=127.19.34.65
--tserver_master_addrs=127.19.34.126:43807
--webserver_port=34963
--webserver_interface=127.19.34.65
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:55:09.352778 3620 log.cc:824] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:09.362910 3620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:55:09.363204 3620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: Bootstrap complete.
I20260619 14:55:09.366484 3620 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5a6df75864d7ea35630d318368cd3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43807 } }
I20260619 14:55:09.366837 3620 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd5a6df75864d7ea35630d318368cd3, State: Initialized, Role: FOLLOWER
I20260619 14:55:09.367276 3620 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5a6df75864d7ea35630d318368cd3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43807 } }
I20260619 14:55:09.367403 3620 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:09.367461 3620 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:09.367540 3620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 2 FOLLOWER]: Advancing to term 3
I20260619 14:55:09.369239 3620 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5a6df75864d7ea35630d318368cd3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43807 } }
I20260619 14:55:09.369545 3620 leader_election.cc:302] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcd5a6df75864d7ea35630d318368cd3; no voters:
I20260619 14:55:09.369794 3620 leader_election.cc:288] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 14:55:09.369889 3624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 14:55:09.370115 3624 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [term 3 LEADER]: Becoming Leader. State: Replica: fcd5a6df75864d7ea35630d318368cd3, State: Running, Role: LEADER
I20260619 14:55:09.370425 3624 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5a6df75864d7ea35630d318368cd3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43807 } }
I20260619 14:55:09.370843 3620 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:09.371764 3625 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fcd5a6df75864d7ea35630d318368cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5a6df75864d7ea35630d318368cd3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43807 } } }
I20260619 14:55:09.371789 3626 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcd5a6df75864d7ea35630d318368cd3. Latest consensus state: current_term: 3 leader_uuid: "fcd5a6df75864d7ea35630d318368cd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcd5a6df75864d7ea35630d318368cd3" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43807 } } }
I20260619 14:55:09.371891 3625 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:09.371891 3626 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:09.372288 3633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:09.376868 3633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:09.377496 3633 catalog_manager.cc:1259] Loaded cluster ID: c55fcd9499784bf28c151330182c5c49
I20260619 14:55:09.377573 3633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:09.379976 3633 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:09.380635 3633 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: Loaded TSK: 0
I20260619 14:55:09.380756 3633 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fcd5a6df75864d7ea35630d318368cd3: Loaded TSK: 1
F20260619 14:55:09.381474 3633 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9a480399ad google::LogMessage::Fail() at ??:0
@ 0x7f9a4803c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f9a480394de google::LogMessage::Flush() at ??:0
@ 0x7f9a4803aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9a4acfd18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f9a4acf5f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f9a5299e7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f9a5297899c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9a52979178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f9a529badf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f9a529d982b std::function<>::operator()() at ??:0
@ 0x7f9a52978b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f9a52979f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9a52976579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f9a529ba640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9a4a28b3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9a4a28bc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f9a4a28d3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9a4a27c512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9a4b8c06db start_thread at ??:0
@ 0x7f9a46b3761f clone at ??:0
W20260619 14:55:09.457810 3622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:09.457968 3622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:09.458001 3622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:09.462095 3622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:09.462131 3622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:09.462177 3622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:55:09.467290 3622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:34725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=34963
--tserver_master_addrs=127.19.34.126:43807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:09.468189 3622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:09.469069 3622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:09.477088 3667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:09.477113 3668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:09.477279 3670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:09.477505 3622 server_base.cc:1034] running on GCE node
I20260619 14:55:09.477902 3622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:09.478605 3622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:09.479801 3622 hybrid_clock.cc:648] HybridClock initialized: now 1781880909479770 us; error 38 us; skew 500 ppm
Jun 19 14:55:09 dist-test-slave-3qdh krb5kdc[2475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880909, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:09.482784 3622 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:55:09.483546 3622 webserver.cc:460] Webserver started at http://127.19.34.65:34963/ using document root <none> and password file <none>
I20260619 14:55:09.484007 3622 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:09.484059 3622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:09.487460 3622 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:09.489411 3677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:09.490509 3622 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:55:09.490706 3622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "3c5dbac576ad47e1adcf37f7518bc5c7"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-3qdh"
server_key: "b36d70874a1c9597c78c8cf0dc43f62e"
server_key_iv: "371c241bdb57a724340565d4a3e94999"
server_key_version: "encryptionkey@0"
I20260619 14:55:09.491200 3622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:09.504015 3622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:09.506696 3622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:09.506876 3622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:09.507364 3622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:55:09.508347 3684 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:55:09.512943 3622 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:55:09.513007 3622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260619 14:55:09.513091 3622 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:55:09.514425 3622 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:55:09.514484 3622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260619 14:55:09.514734 3684 tablet_bootstrap.cc:492] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7: Bootstrap starting.
I20260619 14:55:09.523784 3622 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:34725
I20260619 14:55:09.523967 3791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:34725 every 8 connection(s)
I20260619 14:55:09.524626 3622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:55:09.533484 3684 log.cc:824] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:09.534220 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 3622
I20260619 14:55:09.534631 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:35273
--local_ip_for_outbound_sockets=127.19.34.66
--tserver_master_addrs=127.19.34.126:43807
--webserver_port=36785
--webserver_interface=127.19.34.66
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:55:09.539425 3684 tablet_bootstrap.cc:492] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:55:09.539731 3684 tablet_bootstrap.cc:492] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7: Bootstrap complete.
I20260619 14:55:09.540716 3684 ts_tablet_manager.cc:1397] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.010s
I20260619 14:55:09.542688 3684 raft_consensus.cc:348] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:55:09.543033 3684 raft_consensus.cc:729] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5dbac576ad47e1adcf37f7518bc5c7, State: Initialized, Role: FOLLOWER
I20260619 14:55:09.543450 3684 consensus_queue.cc:260] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:55:09.544250 3684 ts_tablet_manager.cc:1428] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
W20260619 14:55:09.603436 3794 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:09.526268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.65:36637)
0619 14:55:09.526653 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:09.526695 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:09.527335 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:09.602786 (+ 75451us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":118,"threads_started":1}
W20260619 14:55:09.603967 3792 heartbeater.cc:644] Failed to heartbeat to 127.19.34.126:43807 (0 consecutive failures): Network error: Failed to ping master at 127.19.34.126:43807: Client connection negotiation failed: client connection to 127.19.34.126:43807: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:55:09.604699 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:09.604467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.65:44205)
0619 14:55:09.604580 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:09.604642 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 14:55:09.605451 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:09.605248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.65:45075)
0619 14:55:09.605350 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:09.605394 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 14:55:09.605628 3792 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:55:09.644076 3796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:09.644351 3796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:09.644446 3796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:09.647950 3796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:09.648016 3796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:09.648089 3796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:55:09.652601 3796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:35273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=36785
--tserver_master_addrs=127.19.34.126:43807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:09.653434 3796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:09.654256 3796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:09.662303 3802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:09.662333 3803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:09.662333 3805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:09.662465 3796 server_base.cc:1034] running on GCE node
I20260619 14:55:09.662829 3796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:09.663323 3796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:09.664495 3796 hybrid_clock.cc:648] HybridClock initialized: now 1781880909664486 us; error 21 us; skew 500 ppm
Jun 19 14:55:09 dist-test-slave-3qdh krb5kdc[2475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880909, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:09.666734 3796 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:55:09.667392 3796 webserver.cc:460] Webserver started at http://127.19.34.66:36785/ using document root <none> and password file <none>
I20260619 14:55:09.667847 3796 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:09.667902 3796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:09.670923 3796 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:09.672820 3812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:09.673844 3796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:09.673938 3796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "c353bc3ddb754f89a860026be0c983fc"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-3qdh"
server_key: "9c696cfacfd973538cbd0c0c9252494b"
server_key_iv: "8d2c70f7cf838eee77cd60df88fd0e70"
server_key_version: "encryptionkey@0"
I20260619 14:55:09.674400 3796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:09.686928 3796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:09.689280 3796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:09.689399 3796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:09.689813 3796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:55:09.690716 3819 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260619 14:55:09.694177 3819 ts_tablet_manager.cc:1852] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 14:55:09.694547 3819 ts_tablet_manager.cc:1905] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:55:09.698415 3819 ts_tablet_manager.cc:1918] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:55:09.698501 3819 log.cc:1196] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/wal/wals/ed43406a866948c7bc29e3d438d9cd8d
I20260619 14:55:09.698786 3819 ts_tablet_manager.cc:1939] T ed43406a866948c7bc29e3d438d9cd8d P c353bc3ddb754f89a860026be0c983fc: Deleting consensus metadata
I20260619 14:55:09.699023 3796 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 14:55:09.699084 3796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260619 14:55:09.699165 3796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:55:09.699201 3796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:09.707976 3796 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:35273
I20260619 14:55:09.708021 3926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:35273 every 8 connection(s)
I20260619 14:55:09.708755 3796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:55:09.710819 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 3796
I20260619 14:55:09.711264 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:35341
--local_ip_for_outbound_sockets=127.19.34.67
--tserver_master_addrs=127.19.34.126:43807
--webserver_port=37131
--webserver_interface=127.19.34.67
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:38203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:55:09.711643 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:09.710361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.66:39945)
0619 14:55:09.710785 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:09.710915 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":97,"threads_started":1}
W20260619 14:55:09.712026 3927 heartbeater.cc:644] Failed to heartbeat to 127.19.34.126:43807 (0 consecutive failures): Network error: Failed to ping master at 127.19.34.126:43807: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
I20260619 14:55:09.712634 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:09.712382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.66:33743)
0619 14:55:09.712482 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:09.712543 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260619 14:55:09.713217 3929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:09.712991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.66:59597)
0619 14:55:09.713100 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:09.713152 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 14:55:09.713363 3927 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:55:09.809309 3930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:09.809517 3930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:09.809578 3930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:09.816248 3930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:09.816322 3930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:09.816407 3930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:55:09.823686 3930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:38203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:35341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=37131
--tserver_master_addrs=127.19.34.126:43807
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:09.824817 3930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:09.825990 3930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:09.835815 3938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:09.835883 3936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:09.835894 3935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:09.836179 3930 server_base.cc:1034] running on GCE node
I20260619 14:55:09.836522 3930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:09.837006 3930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:09.838172 3930 hybrid_clock.cc:648] HybridClock initialized: now 1781880909838162 us; error 24 us; skew 500 ppm
Jun 19 14:55:09 dist-test-slave-3qdh krb5kdc[2475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880909, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:09.840370 3930 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:55:09.841166 3930 webserver.cc:460] Webserver started at http://127.19.34.67:37131/ using document root <none> and password file <none>
I20260619 14:55:09.841665 3930 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:09.841753 3930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:09.844883 3930 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:55:09.846601 3945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:09.847587 3930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:09.847669 3930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "32ad88aff25b4b078b13d2bbc475dd05"
format_stamp: "Formatted at 2026-06-19 14:54:27 on dist-test-slave-3qdh"
server_key: "a8c7766854e882e498b2a81052b1edc6"
server_key_iv: "79cde5ab4c6758cc0ccde9cc9395cd7c"
server_key_version: "encryptionkey@0"
I20260619 14:55:09.848058 3930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:09.863658 3930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:09.865819 3930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:09.865931 3930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:09.866322 3930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:55:09.866954 3930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:55:09.866992 3930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:09.867023 3930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:55:09.867036 3930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:09.875042 3930 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:35341
I20260619 14:55:09.875062 4058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:35341 every 8 connection(s)
I20260619 14:55:09.875782 3930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:55:09.876853 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 3930
I20260619 14:55:09.879338 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:09.877692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.67:53923)
0619 14:55:09.878221 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:09.878330 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":124,"threads_started":1}
W20260619 14:55:09.879695 4059 heartbeater.cc:644] Failed to heartbeat to 127.19.34.126:43807 (0 consecutive failures): Network error: Failed to ping master at 127.19.34.126:43807: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
I20260619 14:55:09.880187 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:09.879979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.67:48737)
0619 14:55:09.880092 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:09.880141 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 14:55:09.880796 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:09.880549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.67:39993)
0619 14:55:09.880661 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:09.880747 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 14:55:09.880959 4059 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 14:55:10.606434 4063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:10.606057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.65:35267)
0619 14:55:10.606298 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:10.606346 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":62,"threads_started":1}
I20260619 14:55:10.714128 4064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:10.713820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.66:52805)
0619 14:55:10.714019 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:10.714062 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":64,"threads_started":1}
I20260619 14:55:10.881713 4065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:10.881370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.67:47329)
0619 14:55:10.881572 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:10.881630 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":59,"threads_started":1}
I20260619 14:55:11.021549 4066 raft_consensus.cc:482] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:55:11.021754 4066 raft_consensus.cc:504] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:55:11.022738 4066 leader_election.cc:288] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32ad88aff25b4b078b13d2bbc475dd05 (127.19.34.67:35341), c353bc3ddb754f89a860026be0c983fc (127.19.34.66:35273)
Jun 19 14:55:11 dist-test-slave-3qdh krb5kdc[2475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880909, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.66@KRBTEST.COM
Jun 19 14:55:11 dist-test-slave-3qdh krb5kdc[2475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880909, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.67@KRBTEST.COM
I20260619 14:55:11.034793 4063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:11.022938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:35273 (local address 127.19.34.65:42107)
0619 14:55:11.023033 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:11.023046 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:11.023137 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:11.024878 (+ 1741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:11.024883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:11.025357 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:11.025739 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:11.025750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:11.026846 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:11.026848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:11.027683 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:11.027692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:11.028347 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:11.028351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:11.028574 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:11.029153 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:11.029166 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:11.030553 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:11.032796 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:11.032800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:11.032811 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:11.033078 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:11.033383 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:11.033386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:11.033387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:11.033482 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:11.034004 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:11.034006 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:11.034179 (+ 173us) client_negotiation.cc:769] Sending connection context
0619 14:55:11.034347 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 14:55:11.034602 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":113,"spinlock_wait_cycles":15360}
I20260619 14:55:11.034821 4067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:11.023088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:41985 (local address 127.19.34.67:35341)
0619 14:55:11.023362 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:11.023384 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:11.023530 (+ 146us) server_negotiation.cc:407] Connection header received
0619 14:55:11.024141 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:11.024146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:11.024552 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:11.024847 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:11.025878 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:11.026683 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:11.027788 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:11.028206 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:11.028384 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:11.031072 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:11.031076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:11.031084 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:11.031109 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:11.032745 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:11.033170 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:11.033175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:11.033183 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:11.033277 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:11.033559 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:11.033563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:11.033564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:11.033866 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:11.033992 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:11.034436 (+ 444us) server_negotiation.cc:299] Negotiation successful
0619 14:55:11.034662 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":66,"threads_started":1}
I20260619 14:55:11.034979 4068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:11.023088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:42107 (local address 127.19.34.66:35273)
0619 14:55:11.023441 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:11.023455 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:11.023494 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:55:11.024223 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:11.024231 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:11.024578 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:11.024851 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:11.025880 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:11.026699 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:11.027795 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:11.028208 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:11.028389 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:11.030766 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:11.030788 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:11.030797 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:11.030838 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:11.032688 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:11.033197 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:11.033201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:11.033204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:11.033294 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:11.033563 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:11.033567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:11.033569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:11.033924 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:11.034029 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:11.034436 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 14:55:11.034634 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":120,"threads_started":1}
I20260619 14:55:11.034821 4069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:11.022938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:35341 (local address 127.19.34.65:41985)
0619 14:55:11.023433 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:11.023446 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:11.023545 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:11.024877 (+ 1332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:11.024883 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:11.025357 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:11.025738 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:11.025750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:11.026801 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:11.026805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:11.027685 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:11.027693 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:11.028346 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:11.028349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:11.028577 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:11.029201 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:11.029212 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:11.030912 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:11.032841 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:11.032844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:11.032846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:11.033066 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:11.033358 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:11.033362 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:11.033363 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:11.033477 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:11.033971 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:11.033976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:11.034179 (+ 203us) client_negotiation.cc:769] Sending connection context
0619 14:55:11.034347 (+ 168us) client_negotiation.cc:240] Negotiation successful
0619 14:55:11.034672 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":520,"spinlock_wait_cycles":22272,"thread_start_us":59,"threads_started":1}
W20260619 14:55:11.037241 3681 leader_election.cc:341] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c353bc3ddb754f89a860026be0c983fc (127.19.34.66:35273): Not found: Tablet not found: ed43406a866948c7bc29e3d438d9cd8d
W20260619 14:55:11.037398 3679 leader_election.cc:341] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 32ad88aff25b4b078b13d2bbc475dd05 (127.19.34.67:35341): Not found: Tablet not found: ed43406a866948c7bc29e3d438d9cd8d
I20260619 14:55:11.037483 3679 leader_election.cc:302] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5dbac576ad47e1adcf37f7518bc5c7; no voters: 32ad88aff25b4b078b13d2bbc475dd05, c353bc3ddb754f89a860026be0c983fc
I20260619 14:55:11.037740 4066 raft_consensus.cc:2726] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:55:11.607497 4070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:11.607116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.65:57017)
0619 14:55:11.607360 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:11.607410 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":69,"threads_started":1}
I20260619 14:55:11.715080 4071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:11.714745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.66:55849)
0619 14:55:11.714956 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:11.715000 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":61,"threads_started":1}
I20260619 14:55:11.882807 4072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:11.882497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.67:40331)
0619 14:55:11.882690 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:11.882735 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":57,"threads_started":1}
I20260619 14:55:12.608543 4073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:12.608128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.65:57477)
0619 14:55:12.608403 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:12.608451 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":67,"threads_started":1}
I20260619 14:55:12.716040 4074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:12.715685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.66:59447)
0619 14:55:12.715912 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:12.715959 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":79,"threads_started":1}
I20260619 14:55:12.883872 4075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:12.883466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.67:41029)
0619 14:55:12.883743 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:12.883791 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":99,"threads_started":1}
I20260619 14:55:12.934885 4076 raft_consensus.cc:482] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:55:12.935040 4076 raft_consensus.cc:504] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:55:12.935508 4076 leader_election.cc:288] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32ad88aff25b4b078b13d2bbc475dd05 (127.19.34.67:35341), c353bc3ddb754f89a860026be0c983fc (127.19.34.66:35273)
W20260619 14:55:12.936290 3681 leader_election.cc:341] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c353bc3ddb754f89a860026be0c983fc (127.19.34.66:35273): Not found: Tablet not found: ed43406a866948c7bc29e3d438d9cd8d
W20260619 14:55:12.936401 3679 leader_election.cc:341] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 32ad88aff25b4b078b13d2bbc475dd05 (127.19.34.67:35341): Not found: Tablet not found: ed43406a866948c7bc29e3d438d9cd8d
I20260619 14:55:12.936466 3679 leader_election.cc:302] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5dbac576ad47e1adcf37f7518bc5c7; no voters: 32ad88aff25b4b078b13d2bbc475dd05, c353bc3ddb754f89a860026be0c983fc
I20260619 14:55:12.936583 4076 raft_consensus.cc:2726] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:55:13.609535 4077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:13.609176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.65:36917)
0619 14:55:13.609398 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:13.609447 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":65,"threads_started":1}
I20260619 14:55:13.717044 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:13.716671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.66:49125)
0619 14:55:13.716900 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:13.716948 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":64,"threads_started":1}
I20260619 14:55:13.885053 4079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:13.884575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.67:58561)
0619 14:55:13.884862 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:13.884931 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":100,"threads_started":1}
I20260619 14:55:14.610778 4080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:14.610337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.65:52385)
0619 14:55:14.610632 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:14.610688 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":89,"threads_started":1}
I20260619 14:55:14.718046 4081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:14.717703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.66:56861)
0619 14:55:14.717925 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:14.717973 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":77,"threads_started":1}
I20260619 14:55:14.885984 4082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:14.885687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.67:50011)
0619 14:55:14.885872 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:14.885914 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":58,"threads_started":1}
I20260619 14:55:15.046506 4083 raft_consensus.cc:482] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:55:15.046664 4083 raft_consensus.cc:504] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:55:15.047035 4083 leader_election.cc:288] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32ad88aff25b4b078b13d2bbc475dd05 (127.19.34.67:35341), c353bc3ddb754f89a860026be0c983fc (127.19.34.66:35273)
W20260619 14:55:15.047751 3681 leader_election.cc:341] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c353bc3ddb754f89a860026be0c983fc (127.19.34.66:35273): Not found: Tablet not found: ed43406a866948c7bc29e3d438d9cd8d
W20260619 14:55:15.047855 3679 leader_election.cc:341] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 32ad88aff25b4b078b13d2bbc475dd05 (127.19.34.67:35341): Not found: Tablet not found: ed43406a866948c7bc29e3d438d9cd8d
I20260619 14:55:15.047925 3679 leader_election.cc:302] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5dbac576ad47e1adcf37f7518bc5c7; no voters: 32ad88aff25b4b078b13d2bbc475dd05, c353bc3ddb754f89a860026be0c983fc
I20260619 14:55:15.048039 4083 raft_consensus.cc:2726] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:55:15.611915 4084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:15.611557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.65:33721)
0619 14:55:15.611776 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:15.611826 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":64,"threads_started":1}
I20260619 14:55:15.719110 4085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:15.718669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.66:58521)
0619 14:55:15.718993 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:15.719039 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":69,"threads_started":1}
I20260619 14:55:15.887040 4086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:15.886724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.67:44713)
0619 14:55:15.886922 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:15.886969 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":62,"threads_started":1}
I20260619 14:55:16.590018 4087 raft_consensus.cc:482] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:55:16.590137 4087 raft_consensus.cc:504] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:55:16.590565 4087 leader_election.cc:288] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32ad88aff25b4b078b13d2bbc475dd05 (127.19.34.67:35341), c353bc3ddb754f89a860026be0c983fc (127.19.34.66:35273)
W20260619 14:55:16.591251 3679 leader_election.cc:341] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 32ad88aff25b4b078b13d2bbc475dd05 (127.19.34.67:35341): Not found: Tablet not found: ed43406a866948c7bc29e3d438d9cd8d
W20260619 14:55:16.591346 3681 leader_election.cc:341] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c353bc3ddb754f89a860026be0c983fc (127.19.34.66:35273): Not found: Tablet not found: ed43406a866948c7bc29e3d438d9cd8d
I20260619 14:55:16.591411 3681 leader_election.cc:302] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5dbac576ad47e1adcf37f7518bc5c7; no voters: 32ad88aff25b4b078b13d2bbc475dd05, c353bc3ddb754f89a860026be0c983fc
I20260619 14:55:16.591588 4087 raft_consensus.cc:2726] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:55:16.612867 4088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.612559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.65:51015)
0619 14:55:16.612753 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.612795 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":54,"threads_started":1}
I20260619 14:55:16.720161 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.719797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.66:45969)
0619 14:55:16.720042 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.720087 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":58,"threads_started":1}
I20260619 14:55:16.888031 4090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:16.887681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.67:60797)
0619 14:55:16.887919 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:16.887963 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":57,"threads_started":1}
I20260619 14:55:17.613828 4091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:17.613467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.65:60927)
0619 14:55:17.613691 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:17.613740 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260619 14:55:17.721125 4092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:17.720785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.66:52857)
0619 14:55:17.721004 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:17.721050 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":74,"threads_started":1}
I20260619 14:55:17.888968 4093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:17.888636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.67:41139)
0619 14:55:17.888850 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:17.888893 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":78,"threads_started":1}
I20260619 14:55:18.614943 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:18.614594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.65:39147)
0619 14:55:18.614805 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:18.614853 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":65,"threads_started":1}
I20260619 14:55:18.722138 4095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:18.721756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.66:41905)
0619 14:55:18.721992 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:18.722053 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":68,"threads_started":1}
I20260619 14:55:18.889876 4096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:18.889545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.67:38801)
0619 14:55:18.889764 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:18.889808 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":80,"threads_started":1}
I20260619 14:55:19.616089 4097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:19.615734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.65:39721)
0619 14:55:19.615951 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:19.615999 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":67,"threads_started":1}
I20260619 14:55:19.723191 4098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:19.722820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.66:38693)
0619 14:55:19.723050 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:19.723100 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":64,"threads_started":1}
I20260619 14:55:19.890873 4099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:19.890504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.67:48915)
0619 14:55:19.890743 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:19.890792 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":90,"threads_started":1}
I20260619 14:55:20.238341 4100 raft_consensus.cc:482] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:55:20.238487 4100 raft_consensus.cc:504] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32ad88aff25b4b078b13d2bbc475dd05" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35341 } } peers { permanent_uuid: "c353bc3ddb754f89a860026be0c983fc" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 35273 } } peers { permanent_uuid: "3c5dbac576ad47e1adcf37f7518bc5c7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34725 } }
I20260619 14:55:20.238931 4100 leader_election.cc:288] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 32ad88aff25b4b078b13d2bbc475dd05 (127.19.34.67:35341), c353bc3ddb754f89a860026be0c983fc (127.19.34.66:35273)
W20260619 14:55:20.239710 3679 leader_election.cc:341] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 32ad88aff25b4b078b13d2bbc475dd05 (127.19.34.67:35341): Not found: Tablet not found: ed43406a866948c7bc29e3d438d9cd8d
W20260619 14:55:20.239809 3681 leader_election.cc:341] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c353bc3ddb754f89a860026be0c983fc (127.19.34.66:35273): Not found: Tablet not found: ed43406a866948c7bc29e3d438d9cd8d
I20260619 14:55:20.239871 3681 leader_election.cc:302] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5dbac576ad47e1adcf37f7518bc5c7; no voters: 32ad88aff25b4b078b13d2bbc475dd05, c353bc3ddb754f89a860026be0c983fc
I20260619 14:55:20.239996 4100 raft_consensus.cc:2726] T ed43406a866948c7bc29e3d438d9cd8d P 3c5dbac576ad47e1adcf37f7518bc5c7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:55:20.617144 4101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:20.616789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.65:39135)
0619 14:55:20.617009 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:20.617058 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":68,"threads_started":1}
I20260619 14:55:20.724347 4102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:20.723933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.66:59195)
0619 14:55:20.724158 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:20.724212 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":66,"threads_started":1}
I20260619 14:55:20.891909 4103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:20.891543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.67:52059)
0619 14:55:20.891778 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:20.891827 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":78,"threads_started":1}
I20260619 14:55:21.618194 4104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.617833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.65:51873)
0619 14:55:21.618051 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.618098 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":75,"threads_started":1}
I20260619 14:55:21.725308 4105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.724973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.66:49265)
0619 14:55:21.725181 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.725229 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":60,"threads_started":1}
I20260619 14:55:21.892937 4106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:21.892578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.67:39333)
0619 14:55:21.892790 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:21.892838 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":64,"threads_started":1}
I20260619 14:55:22.619215 4107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:22.618823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.65:57483)
0619 14:55:22.619075 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:22.619122 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":94,"threads_started":1}
I20260619 14:55:22.726279 4108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:22.725919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.66:53155)
0619 14:55:22.726143 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:22.726200 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
I20260619 14:55:22.893822 4109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:22.893476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.67:59487)
0619 14:55:22.893700 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:22.893751 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":58,"threads_started":1}
I20260619 14:55:23.620411 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:23.620006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.65:54037)
0619 14:55:23.620252 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:23.620324 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":83,"threads_started":1}
I20260619 14:55:23.727349 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:23.726986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.66:42785)
0619 14:55:23.727215 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:23.727265 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":82,"threads_started":1}
I20260619 14:55:23.894701 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:23.894395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.67:58063)
0619 14:55:23.894579 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:23.894621 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":57,"threads_started":1}
I20260619 14:55:24.621574 4113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:24.621205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.65:54445)
0619 14:55:24.621438 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:24.621488 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":65,"threads_started":1}
I20260619 14:55:24.728526 4114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:24.728139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43807 (local address 127.19.34.66:33781)
0619 14:55:24.728401 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:24.728448 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.126:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":64,"threads_started":1}
I20260619 14:55:24.882454 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 3622
I20260619 14:55:24.887928 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 3796
I20260619 14:55:24.892949 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 3930
2026-06-19T14:55:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59054 ms)
[----------] 30 tests from SecurityITest (227873 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:24 dist-test-slave-3qdh krb5kdc[4117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:24 dist-test-slave-3qdh krb5kdc[4117](info): set up 2 sockets
Jun 19 14:55:24 dist-test-slave-3qdh krb5kdc[4117](info): commencing operation
krb5kdc: starting...
W20260619 14:55:25.887858 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:25 dist-test-slave-3qdh krb5kdc[4117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:55:25.993721 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:40953
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:40953
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:26.095082 4130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:26.095294 4130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:26.095335 4130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:26.099090 4130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:26.099138 4130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:26.099156 4130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:26.099206 4130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:26.099239 4130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:26.103904 4130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:40953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:40953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:26.104745 4130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:26.105630 4130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:26.112972 4136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:26.112988 4138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:26.113237 4130 server_base.cc:1034] running on GCE node
W20260619 14:55:26.112955 4135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:26.113884 4130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:26.114861 4130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:26.116026 4130 hybrid_clock.cc:648] HybridClock initialized: now 1781880926116014 us; error 31 us; skew 500 ppm
Jun 19 14:55:26 dist-test-slave-3qdh krb5kdc[4117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880926, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:26.123695 4130 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:55:26.124435 4130 webserver.cc:460] Webserver started at http://127.19.34.126:38813/ using document root <none> and password file <none>
I20260619 14:55:26.124936 4130 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:26.125022 4130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:26.125228 4130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:26.126995 4130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "e4a0a184f6d2446d984a61c219b0dbbe"
format_stamp: "Formatted at 2026-06-19 14:55:26 on dist-test-slave-3qdh"
server_key: "cfd307b740690bc56d6d4dec51fd460d"
server_key_iv: "539254b5743bc8a4164f4e8c59e60180"
server_key_version: "encryptionkey@0"
I20260619 14:55:26.127526 4130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "e4a0a184f6d2446d984a61c219b0dbbe"
format_stamp: "Formatted at 2026-06-19 14:55:26 on dist-test-slave-3qdh"
server_key: "cfd307b740690bc56d6d4dec51fd460d"
server_key_iv: "539254b5743bc8a4164f4e8c59e60180"
server_key_version: "encryptionkey@0"
I20260619 14:55:26.130728 4130 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:55:26.132790 4145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:26.133872 4130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:26.134001 4130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "e4a0a184f6d2446d984a61c219b0dbbe"
format_stamp: "Formatted at 2026-06-19 14:55:26 on dist-test-slave-3qdh"
server_key: "cfd307b740690bc56d6d4dec51fd460d"
server_key_iv: "539254b5743bc8a4164f4e8c59e60180"
server_key_version: "encryptionkey@0"
I20260619 14:55:26.134104 4130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:26.144094 4130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:26.146768 4130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:26.146970 4130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:26.153648 4130 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:40953
I20260619 14:55:26.153673 4197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:40953 every 8 connection(s)
I20260619 14:55:26.154458 4130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:26.156812 4198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:26.158732 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 4130
I20260619 14:55:26.158879 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:26.159197 19593 external_mini_cluster.cc:1428] Setting key e5f92d9d6a4321ef474767c67bd76c27
I20260619 14:55:26.162223 4198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe: Bootstrap starting.
I20260619 14:55:26.164137 4198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:26.164887 4198 log.cc:824] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:26.166487 4198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe: No bootstrap required, opened a new log
Jun 19 14:55:26 dist-test-slave-3qdh krb5kdc[4117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:55:26.169551 4198 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a0a184f6d2446d984a61c219b0dbbe" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 40953 } }
I20260619 14:55:26.169734 4198 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:26.169803 4198 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4a0a184f6d2446d984a61c219b0dbbe, State: Initialized, Role: FOLLOWER
I20260619 14:55:26.170204 4198 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a0a184f6d2446d984a61c219b0dbbe" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 40953 } }
I20260619 14:55:26.170356 4198 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:26.170423 4198 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:26.170542 4198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:26.171285 4198 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a0a184f6d2446d984a61c219b0dbbe" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 40953 } }
I20260619 14:55:26.171615 4198 leader_election.cc:302] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4a0a184f6d2446d984a61c219b0dbbe; no voters:
I20260619 14:55:26.171969 4198 leader_election.cc:288] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:26.172025 4203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:26.172291 4203 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe [term 1 LEADER]: Becoming Leader. State: Replica: e4a0a184f6d2446d984a61c219b0dbbe, State: Running, Role: LEADER
I20260619 14:55:26.172621 4201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:26.160408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48706 (local address 127.19.34.126:40953)
0619 14:55:26.160759 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:26.160771 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:26.160799 (+ 28us) server_negotiation.cc:407] Connection header received
0619 14:55:26.161443 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:26.161459 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:26.161779 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:26.162074 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:26.162844 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.163730 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.164614 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.165101 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.165272 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:26.167726 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:26.167730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:26.167740 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:26.167761 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:26.169972 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:26.170351 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:26.170356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:26.170361 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:26.170507 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:26.170779 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:26.170782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:26.170783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:26.171089 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:26.171283 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:26.171488 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 14:55:26.171742 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":121,"threads_started":1}
I20260619 14:55:26.172657 4203 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a0a184f6d2446d984a61c219b0dbbe" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 40953 } }
I20260619 14:55:26.173171 4198 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:26.173767 4203 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4a0a184f6d2446d984a61c219b0dbbe. Latest consensus state: current_term: 1 leader_uuid: "e4a0a184f6d2446d984a61c219b0dbbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a0a184f6d2446d984a61c219b0dbbe" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 40953 } } }
I20260619 14:55:26.173893 4203 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:26.174497 4209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:26.176764 4204 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4a0a184f6d2446d984a61c219b0dbbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a0a184f6d2446d984a61c219b0dbbe" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 40953 } } }
I20260619 14:55:26.176859 4204 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:26.177170 4209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:26.183621 4209 catalog_manager.cc:1347] Generated new cluster ID: 8cb07f661c8347729f22b37665778545
I20260619 14:55:26.183679 4209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:26.192176 4209 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:26.192950 4209 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:26.198474 4209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe: Generated new TSK 0
I20260619 14:55:26.199023 4209 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.34.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
I20260619 14:55:26.226444 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:40953
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:26.324473 4224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:26.324673 4224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:26.324707 4224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:26.328189 4224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:26.328233 4224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:26.328328 4224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:55:26.333179 4224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:40953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:26.333972 4224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:26.334760 4224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:26.343078 4232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:26.343204 4224 server_base.cc:1034] running on GCE node
W20260619 14:55:26.343091 4230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:26.343390 4229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:26.343775 4224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:26.344380 4224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:26.345543 4224 hybrid_clock.cc:648] HybridClock initialized: now 1781880926345519 us; error 36 us; skew 500 ppm
Jun 19 14:55:26 dist-test-slave-3qdh krb5kdc[4117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880926, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:26.348088 4224 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:55:26.348894 4224 webserver.cc:460] Webserver started at http://127.19.34.65:35513/ using document root <none> and password file <none>
I20260619 14:55:26.349376 4224 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:26.349433 4224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:26.349665 4224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:26.351408 4224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "e9d117c0b1c2410eba80ca42f8a18687"
format_stamp: "Formatted at 2026-06-19 14:55:26 on dist-test-slave-3qdh"
server_key: "6c83683dee03c3ea4ea60881bf5bf15e"
server_key_iv: "a0916419b50b246c5e69d78f218d8534"
server_key_version: "encryptionkey@0"
I20260619 14:55:26.351939 4224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "e9d117c0b1c2410eba80ca42f8a18687"
format_stamp: "Formatted at 2026-06-19 14:55:26 on dist-test-slave-3qdh"
server_key: "6c83683dee03c3ea4ea60881bf5bf15e"
server_key_iv: "a0916419b50b246c5e69d78f218d8534"
server_key_version: "encryptionkey@0"
I20260619 14:55:26.355176 4224 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260619 14:55:26.357403 4239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:26.358644 4224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:55:26.358798 4224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "e9d117c0b1c2410eba80ca42f8a18687"
format_stamp: "Formatted at 2026-06-19 14:55:26 on dist-test-slave-3qdh"
server_key: "6c83683dee03c3ea4ea60881bf5bf15e"
server_key_iv: "a0916419b50b246c5e69d78f218d8534"
server_key_version: "encryptionkey@0"
I20260619 14:55:26.358898 4224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:26.370944 4224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:26.373948 4224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:26.374110 4224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:26.374586 4224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:55:26.375258 4224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:55:26.375324 4224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:26.375399 4224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:55:26.375434 4224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:26.383436 4224 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:41337
I20260619 14:55:26.383477 4352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:41337 every 8 connection(s)
I20260619 14:55:26.384120 4224 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:55:26.391227 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 4224
I20260619 14:55:26.391351 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:55:26.391649 19593 external_mini_cluster.cc:1428] Setting key 46a94217c429e9c0648c22ab9571db74
Jun 19 14:55:26 dist-test-slave-3qdh krb5kdc[4117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880926, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:55:26.396394 4201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:26.385537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:45203 (local address 127.19.34.126:40953)
0619 14:55:26.385687 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:26.385691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:26.386380 (+ 689us) server_negotiation.cc:407] Connection header received
0619 14:55:26.387478 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:26.387480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:26.387523 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:26.387595 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:26.388556 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.389034 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.390004 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.390198 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.390263 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:26.392596 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:26.392600 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:26.392603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:26.392636 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:26.394265 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:26.394702 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:26.394708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:26.394712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:26.394806 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:26.395243 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:26.395246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:26.395247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:26.395389 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:26.395457 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:26.396015 (+ 558us) server_negotiation.cc:299] Negotiation successful
0619 14:55:26.396152 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 14:55:26.397009 4355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:26.385859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:40953 (local address 127.19.34.65:45203)
0619 14:55:26.386250 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:26.386284 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:26.387312 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:26.387695 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:26.387700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:26.387984 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:26.388426 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:26.388438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.389137 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.389140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:26.389893 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:26.389901 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.390305 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.390307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:26.390420 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:26.390895 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:26.390908 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:26.392440 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:26.394388 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:26.394392 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:26.394400 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:26.394612 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:26.394947 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:26.394952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:26.394954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:26.395153 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:26.395482 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:26.395490 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:26.395765 (+ 275us) client_negotiation.cc:769] Sending connection context
0619 14:55:26.395960 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 14:55:26.396211 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":99,"threads_started":1}
I20260619 14:55:26.398281 4353 heartbeater.cc:347] Connected to a master server at 127.19.34.126:40953
I20260619 14:55:26.398515 4353 heartbeater.cc:464] Registering TS with master...
I20260619 14:55:26.398952 4353 heartbeater.cc:511] Master 127.19.34.126:40953 requested a full tablet report, sending...
I20260619 14:55:26.400257 4162 ts_manager.cc:194] Registered new tserver with Master: e9d117c0b1c2410eba80ca42f8a18687 (127.19.34.65:41337)
WARNING: no policy specified for kudu/127.19.34.66@KRBTEST.COM; defaulting to no policy
I20260619 14:55:26.401363 4162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:45203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
I20260619 14:55:26.417142 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:40953
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:26.513351 4358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:26.513535 4358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:26.513571 4358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:26.517170 4358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:26.517211 4358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:26.517257 4358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:55:26.521607 4358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:40953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:26.522300 4358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:26.523048 4358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:26.531109 4366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:26.531162 4363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:26.531191 4358 server_base.cc:1034] running on GCE node
W20260619 14:55:26.531311 4364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:26.531672 4358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:26.532121 4358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:26.533320 4358 hybrid_clock.cc:648] HybridClock initialized: now 1781880926533297 us; error 42 us; skew 500 ppm
Jun 19 14:55:26 dist-test-slave-3qdh krb5kdc[4117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880926, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:26.535627 4358 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:55:26.536415 4358 webserver.cc:460] Webserver started at http://127.19.34.66:34561/ using document root <none> and password file <none>
I20260619 14:55:26.536888 4358 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:26.536973 4358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:26.537177 4358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:26.538789 4358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "e8d69770702e4359b9b86284d1667d31"
format_stamp: "Formatted at 2026-06-19 14:55:26 on dist-test-slave-3qdh"
server_key: "152eb48642bc58cffb308026930bcffa"
server_key_iv: "63a9a34bfc258cfac1936a2cb8cb2398"
server_key_version: "encryptionkey@0"
I20260619 14:55:26.539287 4358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "e8d69770702e4359b9b86284d1667d31"
format_stamp: "Formatted at 2026-06-19 14:55:26 on dist-test-slave-3qdh"
server_key: "152eb48642bc58cffb308026930bcffa"
server_key_iv: "63a9a34bfc258cfac1936a2cb8cb2398"
server_key_version: "encryptionkey@0"
I20260619 14:55:26.542403 4358 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 14:55:26.544454 4373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:26.545457 4358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:26.545588 4358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "e8d69770702e4359b9b86284d1667d31"
format_stamp: "Formatted at 2026-06-19 14:55:26 on dist-test-slave-3qdh"
server_key: "152eb48642bc58cffb308026930bcffa"
server_key_iv: "63a9a34bfc258cfac1936a2cb8cb2398"
server_key_version: "encryptionkey@0"
I20260619 14:55:26.545683 4358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:26.562522 4358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:26.564817 4358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:26.564990 4358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:26.565428 4358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:55:26.566072 4358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:55:26.566133 4358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:26.566183 4358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:55:26.566226 4358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:26.574121 4358 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:36789
I20260619 14:55:26.574168 4486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:36789 every 8 connection(s)
I20260619 14:55:26.574811 4358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:55:26.581729 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 4358
I20260619 14:55:26.581852 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:55:26.582237 19593 external_mini_cluster.cc:1428] Setting key 3f049eac689672e5d11aaa0cb921e5d0
Jun 19 14:55:26 dist-test-slave-3qdh krb5kdc[4117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880926, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:55:26.586632 4201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:26.576240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:55693 (local address 127.19.34.126:40953)
0619 14:55:26.576389 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:26.576397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:26.577021 (+ 624us) server_negotiation.cc:407] Connection header received
0619 14:55:26.578228 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:26.578231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:26.578284 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:26.578370 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:26.579309 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.579774 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.580885 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.581125 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.581230 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:26.583747 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:26.583752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:26.583758 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:26.583779 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:26.584951 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:26.585378 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:26.585382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:26.585384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:26.585456 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:26.585752 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:26.585756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:26.585757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:26.585908 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:26.586003 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:26.586385 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 14:55:26.586492 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:55:26.587263 4489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:26.576527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:40953 (local address 127.19.34.66:55693)
0619 14:55:26.576891 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:26.576925 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:26.578031 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:26.578512 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:26.578518 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:26.578799 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:26.579178 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:26.579189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.579892 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.579896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:26.580760 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:26.580768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.581274 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.581278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:26.581428 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:26.581942 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:26.581961 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:26.583597 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:26.585054 (+ 1457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:26.585059 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:26.585069 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:26.585285 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:26.585538 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:26.585541 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:26.585543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:26.585663 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:26.586001 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:26.586005 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:26.586192 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 14:55:26.586345 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 14:55:26.586590 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":98,"threads_started":1}
I20260619 14:55:26.588356 4487 heartbeater.cc:347] Connected to a master server at 127.19.34.126:40953
I20260619 14:55:26.588553 4487 heartbeater.cc:464] Registering TS with master...
I20260619 14:55:26.588933 4487 heartbeater.cc:511] Master 127.19.34.126:40953 requested a full tablet report, sending...
I20260619 14:55:26.589847 4162 ts_manager.cc:194] Registered new tserver with Master: e8d69770702e4359b9b86284d1667d31 (127.19.34.66:36789)
I20260619 14:55:26.590528 4162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:55693
WARNING: no policy specified for kudu/127.19.34.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
I20260619 14:55:26.604557 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:40953
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:43605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:26.700057 4492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:26.700235 4492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:26.700271 4492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:26.703905 4492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:26.703948 4492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:26.703994 4492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:55:26.708468 4492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:43605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:40953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:26.709148 4492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:26.709865 4492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:26.718201 4497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:26.718197 4500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:26.718197 4498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:26.718370 4492 server_base.cc:1034] running on GCE node
I20260619 14:55:26.718765 4492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:26.719311 4492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:26.720479 4492 hybrid_clock.cc:648] HybridClock initialized: now 1781880926720456 us; error 40 us; skew 500 ppm
Jun 19 14:55:26 dist-test-slave-3qdh krb5kdc[4117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880926, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:26.722934 4492 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:55:26.723670 4492 webserver.cc:460] Webserver started at http://127.19.34.67:45605/ using document root <none> and password file <none>
I20260619 14:55:26.724170 4492 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:26.724256 4492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:26.724483 4492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:26.726323 4492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "5fecc7bf4ff142109e7cda04a0e69dac"
format_stamp: "Formatted at 2026-06-19 14:55:26 on dist-test-slave-3qdh"
server_key: "ce51b5370342ef2bdff8dbdb32da1d6d"
server_key_iv: "e31a4f84f7e0734686cca7892b10c77d"
server_key_version: "encryptionkey@0"
I20260619 14:55:26.726819 4492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "5fecc7bf4ff142109e7cda04a0e69dac"
format_stamp: "Formatted at 2026-06-19 14:55:26 on dist-test-slave-3qdh"
server_key: "ce51b5370342ef2bdff8dbdb32da1d6d"
server_key_iv: "e31a4f84f7e0734686cca7892b10c77d"
server_key_version: "encryptionkey@0"
I20260619 14:55:26.730024 4492 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:55:26.732018 4507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:26.733191 4492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:55:26.733325 4492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "5fecc7bf4ff142109e7cda04a0e69dac"
format_stamp: "Formatted at 2026-06-19 14:55:26 on dist-test-slave-3qdh"
server_key: "ce51b5370342ef2bdff8dbdb32da1d6d"
server_key_iv: "e31a4f84f7e0734686cca7892b10c77d"
server_key_version: "encryptionkey@0"
I20260619 14:55:26.733433 4492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:26.747900 4492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:26.750487 4492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:26.750675 4492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:26.751168 4492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:55:26.751863 4492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:55:26.751931 4492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:26.751991 4492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:55:26.752036 4492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:26.760106 4492 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:43713
I20260619 14:55:26.760147 4620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:43713 every 8 connection(s)
I20260619 14:55:26.760859 4492 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:55:26.768990 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 4492
I20260619 14:55:26.769109 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
Jun 19 14:55:26 dist-test-slave-3qdh krb5kdc[4117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880926, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:55:26.769383 19593 external_mini_cluster.cc:1428] Setting key e47b9f1d2968c501f5d2f1f118f03747
I20260619 14:55:26.773316 4201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:26.762401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:37785 (local address 127.19.34.126:40953)
0619 14:55:26.762568 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:26.762573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:26.763280 (+ 707us) server_negotiation.cc:407] Connection header received
0619 14:55:26.764509 (+ 1229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:26.764512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:26.764558 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:26.764628 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:26.765606 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.766139 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.767158 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.767357 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.767428 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:26.769835 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:26.769838 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:26.769841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:26.769868 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:26.771372 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:26.771841 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:26.771846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:26.771848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:26.771951 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:26.772285 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:26.772290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:26.772291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:26.772494 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:26.772601 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:26.773040 (+ 439us) server_negotiation.cc:299] Negotiation successful
0619 14:55:26.773154 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:55:26.774025 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:26.762746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:40953 (local address 127.19.34.67:37785)
0619 14:55:26.763135 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:26.763179 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:26.764282 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:26.764764 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:26.764769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:26.765074 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:26.765485 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:26.765497 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.766238 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.766240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:26.767054 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:26.767061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.767468 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.767471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:26.767593 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:26.768125 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:26.768140 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:26.769686 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:26.771478 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:26.771483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:26.771494 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:26.771736 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:26.772050 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:26.772053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:26.772055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:26.772190 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:26.772599 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:26.772605 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:26.772816 (+ 211us) client_negotiation.cc:769] Sending connection context
0619 14:55:26.772986 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 14:55:26.773274 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":106,"threads_started":1}
I20260619 14:55:26.774987 4621 heartbeater.cc:347] Connected to a master server at 127.19.34.126:40953
I20260619 14:55:26.775168 4621 heartbeater.cc:464] Registering TS with master...
I20260619 14:55:26.775561 4621 heartbeater.cc:511] Master 127.19.34.126:40953 requested a full tablet report, sending...
I20260619 14:55:26.776476 4162 ts_manager.cc:194] Registered new tserver with Master: 5fecc7bf4ff142109e7cda04a0e69dac (127.19.34.67:43713)
I20260619 14:55:26.777005 4162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:37785
I20260619 14:55:26.782864 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:55:26.791565 4201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:26.784725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39570 (local address 127.19.34.126:40953)
0619 14:55:26.784858 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:26.784862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:26.785020 (+ 158us) server_negotiation.cc:407] Connection header received
0619 14:55:26.785161 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:26.785163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:26.785208 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:26.785301 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:26.785937 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.786422 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.787229 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.787438 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.787543 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:26.788888 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:26.788892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:26.788894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:26.788924 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:26.790030 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:26.790431 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:26.790435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:26.790437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:26.790510 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:26.790766 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:26.790771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:26.790772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:26.790960 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:26.791090 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:26.791318 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 14:55:26.791424 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:55:26.793907 4162 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:55:26.796646 4162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:55:26.809924 4632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:26.803882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:43713 (local address 127.0.0.1:52554)
0619 14:55:26.804597 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:26.804852 (+ 255us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:26.804964 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:26.805892 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:26.805895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:26.805911 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:26.806008 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:26.806014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.807326 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.807341 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:26.808510 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:26.808521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.809454 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.809457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:26.809552 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:26.809593 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 14:55:26.809683 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 14:55:26.809786 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":5,"spinlock_wait_cycles":2816,"thread_start_us":55,"threads_started":1}
I20260619 14:55:26.810148 4636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:26.804568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.19.34.66:36789)
0619 14:55:26.805197 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:26.805207 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:26.805344 (+ 137us) server_negotiation.cc:407] Connection header received
0619 14:55:26.805446 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:26.805453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:26.805617 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:26.805770 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:26.806172 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.807228 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.808717 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.809479 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.809573 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:26.809693 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:26.809889 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 14:55:26.809974 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":128,"threads_started":1}
I20260619 14:55:26.810285 4635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:26.804499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52554 (local address 127.19.34.67:43713)
0619 14:55:26.805348 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:26.805357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:26.805376 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:55:26.805437 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:26.805442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:26.805613 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:26.805749 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:26.806138 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.807181 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.808637 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.809335 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.809443 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:26.809558 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:26.810016 (+ 458us) server_negotiation.cc:299] Negotiation successful
0619 14:55:26.810105 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":98,"threads_started":1}
I20260619 14:55:26.810524 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:26.804883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:36789 (local address 127.0.0.1:60532)
0619 14:55:26.805242 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:26.805255 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:26.805345 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:26.805927 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:26.805930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:26.805947 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:26.806026 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:26.806031 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.807580 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.807583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:26.808598 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:26.808606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.809656 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.809662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:26.809754 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:26.809772 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:55:26.809836 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 14:55:26.809887 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":70,"threads_started":1}
I20260619 14:55:26.810947 4633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:26.804082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:41337 (local address 127.0.0.1:57564)
0619 14:55:26.804825 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:26.804852 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:26.804964 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:26.805513 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:26.805518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:26.805536 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:26.805636 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:26.805642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.807067 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.807070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:26.808771 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:26.808801 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.809912 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.809915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:26.810562 (+ 647us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:26.810605 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 14:55:26.810714 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 14:55:26.810791 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":298,"spinlock_wait_cycles":6144,"thread_start_us":78,"threads_started":1}
I20260619 14:55:26.811451 4634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:26.804261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.19.34.65:41337)
0619 14:55:26.804697 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:26.804710 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:26.805096 (+ 386us) server_negotiation.cc:407] Connection header received
0619 14:55:26.805192 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:26.805197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:26.805389 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:26.805530 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:26.805745 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.806919 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.808935 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.809627 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.809733 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:26.809856 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:26.811196 (+ 1340us) server_negotiation.cc:299] Negotiation successful
0619 14:55:26.811295 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":135,"threads_started":1}
I20260619 14:55:26.814126 4287 tablet_service.cc:1467] Processing CreateTablet for tablet 9326db02a40d43238e2ba5c8e2b9af8e (DEFAULT_TABLE table=test-table [id=387b7b9dbbf846b087f6dbe47b2018e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:55:26.814138 4421 tablet_service.cc:1467] Processing CreateTablet for tablet 9326db02a40d43238e2ba5c8e2b9af8e (DEFAULT_TABLE table=test-table [id=387b7b9dbbf846b087f6dbe47b2018e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:55:26.815205 4287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9326db02a40d43238e2ba5c8e2b9af8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:26.815256 4421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9326db02a40d43238e2ba5c8e2b9af8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:26.815135 4555 tablet_service.cc:1467] Processing CreateTablet for tablet 9326db02a40d43238e2ba5c8e2b9af8e (DEFAULT_TABLE table=test-table [id=387b7b9dbbf846b087f6dbe47b2018e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:55:26.816107 4555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9326db02a40d43238e2ba5c8e2b9af8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:26.823091 4640 tablet_bootstrap.cc:492] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31: Bootstrap starting.
I20260619 14:55:26.823685 4638 tablet_bootstrap.cc:492] T 9326db02a40d43238e2ba5c8e2b9af8e P e9d117c0b1c2410eba80ca42f8a18687: Bootstrap starting.
I20260619 14:55:26.824616 4639 tablet_bootstrap.cc:492] T 9326db02a40d43238e2ba5c8e2b9af8e P 5fecc7bf4ff142109e7cda04a0e69dac: Bootstrap starting.
I20260619 14:55:26.825518 4640 tablet_bootstrap.cc:654] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:26.826088 4638 tablet_bootstrap.cc:654] T 9326db02a40d43238e2ba5c8e2b9af8e P e9d117c0b1c2410eba80ca42f8a18687: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:26.826241 4640 log.cc:824] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:26.826746 4639 tablet_bootstrap.cc:654] T 9326db02a40d43238e2ba5c8e2b9af8e P 5fecc7bf4ff142109e7cda04a0e69dac: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:26.826929 4638 log.cc:824] T 9326db02a40d43238e2ba5c8e2b9af8e P e9d117c0b1c2410eba80ca42f8a18687: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:26.827885 4639 log.cc:824] T 9326db02a40d43238e2ba5c8e2b9af8e P 5fecc7bf4ff142109e7cda04a0e69dac: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:26.828269 4640 tablet_bootstrap.cc:492] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31: No bootstrap required, opened a new log
I20260619 14:55:26.828488 4640 ts_tablet_manager.cc:1397] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260619 14:55:26.829572 4638 tablet_bootstrap.cc:492] T 9326db02a40d43238e2ba5c8e2b9af8e P e9d117c0b1c2410eba80ca42f8a18687: No bootstrap required, opened a new log
I20260619 14:55:26.829671 4639 tablet_bootstrap.cc:492] T 9326db02a40d43238e2ba5c8e2b9af8e P 5fecc7bf4ff142109e7cda04a0e69dac: No bootstrap required, opened a new log
I20260619 14:55:26.829783 4638 ts_tablet_manager.cc:1397] T 9326db02a40d43238e2ba5c8e2b9af8e P e9d117c0b1c2410eba80ca42f8a18687: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:55:26.829831 4639 ts_tablet_manager.cc:1397] T 9326db02a40d43238e2ba5c8e2b9af8e P 5fecc7bf4ff142109e7cda04a0e69dac: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:55:26.832194 4640 raft_consensus.cc:348] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fecc7bf4ff142109e7cda04a0e69dac" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 43713 } } peers { permanent_uuid: "e9d117c0b1c2410eba80ca42f8a18687" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41337 } } peers { permanent_uuid: "e8d69770702e4359b9b86284d1667d31" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 36789 } }
I20260619 14:55:26.832226 4638 raft_consensus.cc:348] T 9326db02a40d43238e2ba5c8e2b9af8e P e9d117c0b1c2410eba80ca42f8a18687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fecc7bf4ff142109e7cda04a0e69dac" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 43713 } } peers { permanent_uuid: "e9d117c0b1c2410eba80ca42f8a18687" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41337 } } peers { permanent_uuid: "e8d69770702e4359b9b86284d1667d31" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 36789 } }
I20260619 14:55:26.832475 4640 raft_consensus.cc:374] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:26.832507 4638 raft_consensus.cc:374] T 9326db02a40d43238e2ba5c8e2b9af8e P e9d117c0b1c2410eba80ca42f8a18687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:26.832530 4640 raft_consensus.cc:729] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d69770702e4359b9b86284d1667d31, State: Initialized, Role: FOLLOWER
I20260619 14:55:26.832556 4638 raft_consensus.cc:729] T 9326db02a40d43238e2ba5c8e2b9af8e P e9d117c0b1c2410eba80ca42f8a18687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9d117c0b1c2410eba80ca42f8a18687, State: Initialized, Role: FOLLOWER
I20260619 14:55:26.832948 4638 consensus_queue.cc:260] T 9326db02a40d43238e2ba5c8e2b9af8e P e9d117c0b1c2410eba80ca42f8a18687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fecc7bf4ff142109e7cda04a0e69dac" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 43713 } } peers { permanent_uuid: "e9d117c0b1c2410eba80ca42f8a18687" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41337 } } peers { permanent_uuid: "e8d69770702e4359b9b86284d1667d31" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 36789 } }
I20260619 14:55:26.832955 4640 consensus_queue.cc:260] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fecc7bf4ff142109e7cda04a0e69dac" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 43713 } } peers { permanent_uuid: "e9d117c0b1c2410eba80ca42f8a18687" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41337 } } peers { permanent_uuid: "e8d69770702e4359b9b86284d1667d31" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 36789 } }
I20260619 14:55:26.833405 4639 raft_consensus.cc:348] T 9326db02a40d43238e2ba5c8e2b9af8e P 5fecc7bf4ff142109e7cda04a0e69dac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fecc7bf4ff142109e7cda04a0e69dac" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 43713 } } peers { permanent_uuid: "e9d117c0b1c2410eba80ca42f8a18687" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41337 } } peers { permanent_uuid: "e8d69770702e4359b9b86284d1667d31" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 36789 } }
I20260619 14:55:26.833622 4639 raft_consensus.cc:374] T 9326db02a40d43238e2ba5c8e2b9af8e P 5fecc7bf4ff142109e7cda04a0e69dac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:26.833668 4639 raft_consensus.cc:729] T 9326db02a40d43238e2ba5c8e2b9af8e P 5fecc7bf4ff142109e7cda04a0e69dac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fecc7bf4ff142109e7cda04a0e69dac, State: Initialized, Role: FOLLOWER
I20260619 14:55:26.833873 4638 ts_tablet_manager.cc:1428] T 9326db02a40d43238e2ba5c8e2b9af8e P e9d117c0b1c2410eba80ca42f8a18687: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260619 14:55:26.833880 4640 ts_tablet_manager.cc:1428] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260619 14:55:26.834056 4639 consensus_queue.cc:260] T 9326db02a40d43238e2ba5c8e2b9af8e P 5fecc7bf4ff142109e7cda04a0e69dac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fecc7bf4ff142109e7cda04a0e69dac" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 43713 } } peers { permanent_uuid: "e9d117c0b1c2410eba80ca42f8a18687" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41337 } } peers { permanent_uuid: "e8d69770702e4359b9b86284d1667d31" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 36789 } }
I20260619 14:55:26.834239 4487 heartbeater.cc:503] Master 127.19.34.126:40953 was elected leader, sending a full tablet report...
I20260619 14:55:26.834447 4353 heartbeater.cc:503] Master 127.19.34.126:40953 was elected leader, sending a full tablet report...
I20260619 14:55:26.834619 4621 heartbeater.cc:503] Master 127.19.34.126:40953 was elected leader, sending a full tablet report...
I20260619 14:55:26.834841 4639 ts_tablet_manager.cc:1428] T 9326db02a40d43238e2ba5c8e2b9af8e P 5fecc7bf4ff142109e7cda04a0e69dac: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260619 14:55:26.885071 4354 tablet.cc:2369] T 9326db02a40d43238e2ba5c8e2b9af8e P e9d117c0b1c2410eba80ca42f8a18687: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:55:26.907521 4645 raft_consensus.cc:482] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:55:26.907706 4645 raft_consensus.cc:504] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fecc7bf4ff142109e7cda04a0e69dac" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 43713 } } peers { permanent_uuid: "e9d117c0b1c2410eba80ca42f8a18687" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41337 } } peers { permanent_uuid: "e8d69770702e4359b9b86284d1667d31" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 36789 } }
I20260619 14:55:26.908639 4645 leader_election.cc:288] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fecc7bf4ff142109e7cda04a0e69dac (127.19.34.67:43713), e9d117c0b1c2410eba80ca42f8a18687 (127.19.34.65:41337)
I20260619 14:55:26.912818 4634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:26.908965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:59247 (local address 127.19.34.65:41337)
0619 14:55:26.909057 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:26.909061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:26.909178 (+ 117us) server_negotiation.cc:407] Connection header received
0619 14:55:26.909350 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:26.909353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:26.909391 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:26.909457 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:26.909708 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.910305 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.911537 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.912109 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.912268 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:26.912348 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:26.912626 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 14:55:26.912697 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:55:26.912818 4635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:26.908856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:58403 (local address 127.19.34.67:43713)
0619 14:55:26.908965 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:26.908969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:26.909129 (+ 160us) server_negotiation.cc:407] Connection header received
0619 14:55:26.909167 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:26.909169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:26.909199 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:26.909244 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:26.909523 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.910188 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.911537 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.912086 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.912267 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:26.912348 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:26.912626 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 14:55:26.912697 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:55:26.912817 4489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:26.908806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:43713 (local address 127.19.34.66:58403)
0619 14:55:26.908962 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:26.908985 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:26.909068 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:26.909274 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:26.909279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:26.909296 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:26.909388 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:26.909391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.910280 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.910282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:26.911395 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:26.911404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.912254 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.912258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:26.912465 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:26.912514 (+ 49us) client_negotiation.cc:769] Sending connection context
0619 14:55:26.912591 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 14:55:26.912665 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":75}
I20260619 14:55:26.912824 4647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:26.908820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:41337 (local address 127.19.34.66:59247)
0619 14:55:26.909094 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:26.909112 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:26.909208 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:26.909489 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:26.909494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:26.909522 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:26.909610 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:26.909614 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.910385 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.910388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:26.911395 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:26.911404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.912254 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.912258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:26.912468 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:26.912512 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 14:55:26.912591 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 14:55:26.912665 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":107,"thread_start_us":97,"threads_started":1}
I20260619 14:55:26.913393 4307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9326db02a40d43238e2ba5c8e2b9af8e" candidate_uuid: "e8d69770702e4359b9b86284d1667d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d117c0b1c2410eba80ca42f8a18687" is_pre_election: true
I20260619 14:55:26.913642 4307 raft_consensus.cc:2454] T 9326db02a40d43238e2ba5c8e2b9af8e P e9d117c0b1c2410eba80ca42f8a18687 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d69770702e4359b9b86284d1667d31 in term 0.
I20260619 14:55:26.913646 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9326db02a40d43238e2ba5c8e2b9af8e" candidate_uuid: "e8d69770702e4359b9b86284d1667d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fecc7bf4ff142109e7cda04a0e69dac" is_pre_election: true
I20260619 14:55:26.913910 4575 raft_consensus.cc:2454] T 9326db02a40d43238e2ba5c8e2b9af8e P 5fecc7bf4ff142109e7cda04a0e69dac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d69770702e4359b9b86284d1667d31 in term 0.
I20260619 14:55:26.914052 4376 leader_election.cc:302] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8d69770702e4359b9b86284d1667d31, e9d117c0b1c2410eba80ca42f8a18687; no voters:
I20260619 14:55:26.914309 4645 raft_consensus.cc:2781] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:55:26.914377 4645 raft_consensus.cc:482] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:55:26.914413 4645 raft_consensus.cc:3037] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:26.915252 4645 raft_consensus.cc:504] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fecc7bf4ff142109e7cda04a0e69dac" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 43713 } } peers { permanent_uuid: "e9d117c0b1c2410eba80ca42f8a18687" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41337 } } peers { permanent_uuid: "e8d69770702e4359b9b86284d1667d31" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 36789 } }
I20260619 14:55:26.915496 4645 leader_election.cc:288] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 [CANDIDATE]: Term 1 election: Requested vote from peers 5fecc7bf4ff142109e7cda04a0e69dac (127.19.34.67:43713), e9d117c0b1c2410eba80ca42f8a18687 (127.19.34.65:41337)
I20260619 14:55:26.915792 4307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9326db02a40d43238e2ba5c8e2b9af8e" candidate_uuid: "e8d69770702e4359b9b86284d1667d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9d117c0b1c2410eba80ca42f8a18687"
I20260619 14:55:26.915794 4575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9326db02a40d43238e2ba5c8e2b9af8e" candidate_uuid: "e8d69770702e4359b9b86284d1667d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fecc7bf4ff142109e7cda04a0e69dac"
I20260619 14:55:26.915895 4307 raft_consensus.cc:3037] T 9326db02a40d43238e2ba5c8e2b9af8e P e9d117c0b1c2410eba80ca42f8a18687 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:26.915925 4575 raft_consensus.cc:3037] T 9326db02a40d43238e2ba5c8e2b9af8e P 5fecc7bf4ff142109e7cda04a0e69dac [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:26.916837 4307 raft_consensus.cc:2454] T 9326db02a40d43238e2ba5c8e2b9af8e P e9d117c0b1c2410eba80ca42f8a18687 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d69770702e4359b9b86284d1667d31 in term 1.
I20260619 14:55:26.916982 4575 raft_consensus.cc:2454] T 9326db02a40d43238e2ba5c8e2b9af8e P 5fecc7bf4ff142109e7cda04a0e69dac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d69770702e4359b9b86284d1667d31 in term 1.
I20260619 14:55:26.917136 4376 leader_election.cc:302] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e8d69770702e4359b9b86284d1667d31, e9d117c0b1c2410eba80ca42f8a18687; no voters:
I20260619 14:55:26.917277 4645 raft_consensus.cc:2781] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:26.917456 4645 raft_consensus.cc:686] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 [term 1 LEADER]: Becoming Leader. State: Replica: e8d69770702e4359b9b86284d1667d31, State: Running, Role: LEADER
I20260619 14:55:26.917716 4645 consensus_queue.cc:237] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fecc7bf4ff142109e7cda04a0e69dac" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 43713 } } peers { permanent_uuid: "e9d117c0b1c2410eba80ca42f8a18687" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41337 } } peers { permanent_uuid: "e8d69770702e4359b9b86284d1667d31" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 36789 } }
I20260619 14:55:26.920104 4161 catalog_manager.cc:5495] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8d69770702e4359b9b86284d1667d31 (127.19.34.66). New cstate: current_term: 1 leader_uuid: "e8d69770702e4359b9b86284d1667d31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fecc7bf4ff142109e7cda04a0e69dac" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 43713 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9d117c0b1c2410eba80ca42f8a18687" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8d69770702e4359b9b86284d1667d31" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 36789 } health_report { overall_health: HEALTHY } } }
I20260619 14:55:26.929095 4636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:26.926151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.19.34.66:36789)
0619 14:55:26.926252 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:26.926254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:26.926327 (+ 73us) server_negotiation.cc:407] Connection header received
0619 14:55:26.926428 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:26.926430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:26.926470 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:26.926543 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:26.926738 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.927161 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.927874 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:26.928195 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:26.928265 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:26.928533 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:26.928835 (+ 302us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:26.928914 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:26.928967 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 14:55:26.929004 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:55:26.933691 4575 raft_consensus.cc:1261] T 9326db02a40d43238e2ba5c8e2b9af8e P 5fecc7bf4ff142109e7cda04a0e69dac [term 1 FOLLOWER]: Refusing update from remote peer e8d69770702e4359b9b86284d1667d31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:55:26.933691 4307 raft_consensus.cc:1261] T 9326db02a40d43238e2ba5c8e2b9af8e P e9d117c0b1c2410eba80ca42f8a18687 [term 1 FOLLOWER]: Refusing update from remote peer e8d69770702e4359b9b86284d1667d31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:55:26.934262 4648 consensus_queue.cc:1035] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9d117c0b1c2410eba80ca42f8a18687" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:55:26.934408 4645 consensus_queue.cc:1035] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fecc7bf4ff142109e7cda04a0e69dac" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 43713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:55:26.947103 4162 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:55:26.947234 4162 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:55:26.949128 4162 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4a0a184f6d2446d984a61c219b0dbbe: Sending DeleteTablet for 3 replicas of tablet 9326db02a40d43238e2ba5c8e2b9af8e
I20260619 14:55:26.949695 4555 tablet_service.cc:1514] Processing DeleteTablet for tablet 9326db02a40d43238e2ba5c8e2b9af8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:55:26 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:52554
I20260619 14:55:26.949695 4287 tablet_service.cc:1514] Processing DeleteTablet for tablet 9326db02a40d43238e2ba5c8e2b9af8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:55:26 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:57564
I20260619 14:55:26.950071 4661 tablet_replica.cc:330] T 9326db02a40d43238e2ba5c8e2b9af8e P e9d117c0b1c2410eba80ca42f8a18687: stopping tablet replica
I20260619 14:55:26.950073 4662 tablet_replica.cc:330] T 9326db02a40d43238e2ba5c8e2b9af8e P 5fecc7bf4ff142109e7cda04a0e69dac: stopping tablet replica
I20260619 14:55:26.950347 4661 raft_consensus.cc:2229] T 9326db02a40d43238e2ba5c8e2b9af8e P e9d117c0b1c2410eba80ca42f8a18687 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:55:26.950307 4421 tablet_service.cc:1514] Processing DeleteTablet for tablet 9326db02a40d43238e2ba5c8e2b9af8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:55:26 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:60532
I20260619 14:55:26.950453 4662 raft_consensus.cc:2229] T 9326db02a40d43238e2ba5c8e2b9af8e P 5fecc7bf4ff142109e7cda04a0e69dac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:55:26.950579 4661 raft_consensus.cc:2258] T 9326db02a40d43238e2ba5c8e2b9af8e P e9d117c0b1c2410eba80ca42f8a18687 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:55:26.950770 4662 raft_consensus.cc:2258] T 9326db02a40d43238e2ba5c8e2b9af8e P 5fecc7bf4ff142109e7cda04a0e69dac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:55:26.951253 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 4224
I20260619 14:55:26.953621 4662 ts_tablet_manager.cc:1905] T 9326db02a40d43238e2ba5c8e2b9af8e P 5fecc7bf4ff142109e7cda04a0e69dac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:55:26.953790 4663 tablet_replica.cc:330] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31: stopping tablet replica
I20260619 14:55:26.954017 4663 raft_consensus.cc:2229] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:55:26.954272 4663 raft_consensus.cc:2258] T 9326db02a40d43238e2ba5c8e2b9af8e P e8d69770702e4359b9b86284d1667d31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:55:26.956262 4662 ts_tablet_manager.cc:1918] T 9326db02a40d43238e2ba5c8e2b9af8e P 5fecc7bf4ff142109e7cda04a0e69dac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:55:26.956365 4662 log.cc:1196] T 9326db02a40d43238e2ba5c8e2b9af8e P 5fecc7bf4ff142109e7cda04a0e69dac: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/9326db02a40d43238e2ba5c8e2b9af8e
I20260619 14:55:26.956686 4662 ts_tablet_manager.cc:1939] T 9326db02a40d43238e2ba5c8e2b9af8e P 5fecc7bf4ff142109e7cda04a0e69dac: Deleting consensus metadata
W20260619 14:55:26.957409 4148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.34.65:41337 (error 108)
I20260619 14:55:26.957916 4146 catalog_manager.cc:4841] TS 5fecc7bf4ff142109e7cda04a0e69dac (127.19.34.67:43713): tablet 9326db02a40d43238e2ba5c8e2b9af8e (table test-table [id=387b7b9dbbf846b087f6dbe47b2018e6]) successfully deleted
W20260619 14:55:26.958500 4633 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:26.958025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:41337 (local address 127.0.0.1:57570)
0619 14:55:26.958167 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:26.958177 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:26.958230 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:26.958399 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.65:41337: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":74}
W20260619 14:55:26.958637 4148 catalog_manager.cc:4584] TS e9d117c0b1c2410eba80ca42f8a18687 (127.19.34.65:41337): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9326db02a40d43238e2ba5c8e2b9af8e: Network error: Client connection negotiation failed: client connection to 127.19.34.65:41337: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:55:26.958804 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 4358
I20260619 14:55:26.963469 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 4492
I20260619 14:55:26.964203 4633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:26.963564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:36789 (local address 127.0.0.1:60556)
0619 14:55:26.964076 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:26.964148 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.66:36789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454}
I20260619 14:55:26.968791 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 4130
2026-06-19T14:55:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2075 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T14:55:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:26Z Disabled control of system clock
I20260619 14:55:26.997748 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:44735
--webserver_interface=127.19.34.126
--webserver_port=0
--builtin_ntp_servers=127.19.34.84:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:44735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:55:27.094420 4668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:27.094620 4668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:27.094650 4668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:27.098374 4668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:27.098419 4668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:27.098433 4668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:27.098446 4668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:27.098459 4668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:27.102456 4668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:44735
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:44735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:27.103243 4668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:27.104310 4668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:55:27.110992 4668 server_base.cc:1034] running on GCE node
W20260619 14:55:27.111126 4674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:27.110956 4676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:27.110957 4673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:27.111732 4668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:27.112733 4668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:27.113909 4668 hybrid_clock.cc:648] HybridClock initialized: now 1781880927113900 us; error 33 us; skew 500 ppm
I20260619 14:55:27.115397 4668 webserver.cc:460] Webserver started at http://127.19.34.126:44913/ using document root <none> and password file <none>
I20260619 14:55:27.115867 4668 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:27.115948 4668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:27.116142 4668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:27.117828 4668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "471d8f7c2ddb4616841fb0769f62ff3c"
format_stamp: "Formatted at 2026-06-19 14:55:27 on dist-test-slave-3qdh"
server_key: "de9e8da14d6e6d189e35d66127a46444"
server_key_iv: "1338ef377c73c74ed637f8be8cae4617"
server_key_version: "encryptionkey@0"
I20260619 14:55:27.118304 4668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "471d8f7c2ddb4616841fb0769f62ff3c"
format_stamp: "Formatted at 2026-06-19 14:55:27 on dist-test-slave-3qdh"
server_key: "de9e8da14d6e6d189e35d66127a46444"
server_key_iv: "1338ef377c73c74ed637f8be8cae4617"
server_key_version: "encryptionkey@0"
I20260619 14:55:27.121282 4668 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:55:27.123265 4682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:27.124359 4668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:55:27.124471 4668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "471d8f7c2ddb4616841fb0769f62ff3c"
format_stamp: "Formatted at 2026-06-19 14:55:27 on dist-test-slave-3qdh"
server_key: "de9e8da14d6e6d189e35d66127a46444"
server_key_iv: "1338ef377c73c74ed637f8be8cae4617"
server_key_version: "encryptionkey@0"
I20260619 14:55:27.124575 4668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:27.140998 4668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:27.141491 4668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:27.141644 4668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:27.147238 4668 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:44735
I20260619 14:55:27.147267 4734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:44735 every 8 connection(s)
I20260619 14:55:27.148006 4668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:27.150857 4735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:27.152200 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 4668
I20260619 14:55:27.152339 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:27.152648 19593 external_mini_cluster.cc:1428] Setting key f4b4a78b67444732b41ffc4b0d8e4e6e
I20260619 14:55:27.156198 4735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c: Bootstrap starting.
I20260619 14:55:27.157989 4735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:27.158666 4735 log.cc:824] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:27.160364 4735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c: No bootstrap required, opened a new log
I20260619 14:55:27.160645 4738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:27.153872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35886 (local address 127.19.34.126:44735)
0619 14:55:27.154396 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:27.154409 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:27.154437 (+ 28us) server_negotiation.cc:407] Connection header received
0619 14:55:27.155004 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:27.155016 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:27.155323 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:27.155602 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:27.155965 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.156884 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.157701 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.158173 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.158323 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:27.159123 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:27.159127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:27.159137 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:27.159445 (+ 308us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:27.159490 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:27.159507 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:27.159604 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:27.159744 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 14:55:27.159967 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":108,"threads_started":1}
I20260619 14:55:27.163756 4735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471d8f7c2ddb4616841fb0769f62ff3c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44735 } }
I20260619 14:55:27.163962 4735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:27.164044 4735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 471d8f7c2ddb4616841fb0769f62ff3c, State: Initialized, Role: FOLLOWER
I20260619 14:55:27.164451 4735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471d8f7c2ddb4616841fb0769f62ff3c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44735 } }
I20260619 14:55:27.164583 4735 raft_consensus.cc:388] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:27.164664 4735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:27.164750 4735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:27.165481 4735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471d8f7c2ddb4616841fb0769f62ff3c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44735 } }
I20260619 14:55:27.165890 4735 leader_election.cc:302] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 471d8f7c2ddb4616841fb0769f62ff3c; no voters:
I20260619 14:55:27.166249 4735 leader_election.cc:288] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:27.166410 4740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:27.166628 4740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c [term 1 LEADER]: Becoming Leader. State: Replica: 471d8f7c2ddb4616841fb0769f62ff3c, State: Running, Role: LEADER
I20260619 14:55:27.166879 4740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471d8f7c2ddb4616841fb0769f62ff3c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44735 } }
I20260619 14:55:27.167335 4735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:27.168387 4741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "471d8f7c2ddb4616841fb0769f62ff3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471d8f7c2ddb4616841fb0769f62ff3c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44735 } } }
I20260619 14:55:27.168535 4741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:27.168802 4742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 471d8f7c2ddb4616841fb0769f62ff3c. Latest consensus state: current_term: 1 leader_uuid: "471d8f7c2ddb4616841fb0769f62ff3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "471d8f7c2ddb4616841fb0769f62ff3c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 44735 } } }
I20260619 14:55:27.168898 4742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:27.169531 4749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:27.172143 4749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:27.176635 4749 catalog_manager.cc:1347] Generated new cluster ID: b5cecc15fb7947da9d856fa60131464c
I20260619 14:55:27.176687 4749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:27.184118 4749 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:27.184890 4749 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:27.192759 4749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c: Generated new TSK 0
I20260619 14:55:27.193274 4749 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:27.202379 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44735
--builtin_ntp_servers=127.19.34.84:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:55:27.296829 4759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:27.297029 4759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:27.297062 4759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:27.300628 4759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:27.300666 4759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:27.300712 4759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:55:27.304901 4759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:27.305645 4759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:27.306660 4759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:27.314853 4767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:27.314903 4764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:27.314939 4759 server_base.cc:1034] running on GCE node
W20260619 14:55:27.314857 4765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:27.315449 4759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:27.315968 4759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:27.317147 4759 hybrid_clock.cc:648] HybridClock initialized: now 1781880927317121 us; error 36 us; skew 500 ppm
I20260619 14:55:27.318746 4759 webserver.cc:460] Webserver started at http://127.19.34.65:46585/ using document root <none> and password file <none>
I20260619 14:55:27.319240 4759 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:27.319324 4759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:27.319517 4759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:27.321318 4759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "e86c78417b4246f3a01265f289734539"
format_stamp: "Formatted at 2026-06-19 14:55:27 on dist-test-slave-3qdh"
server_key: "f6cb72d1bb8cfbe09554d9f02aa0a2db"
server_key_iv: "251a6e145b845cd8d714ea24c35bee1c"
server_key_version: "encryptionkey@0"
I20260619 14:55:27.321818 4759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "e86c78417b4246f3a01265f289734539"
format_stamp: "Formatted at 2026-06-19 14:55:27 on dist-test-slave-3qdh"
server_key: "f6cb72d1bb8cfbe09554d9f02aa0a2db"
server_key_iv: "251a6e145b845cd8d714ea24c35bee1c"
server_key_version: "encryptionkey@0"
I20260619 14:55:27.325129 4759 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:55:27.327050 4773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:27.328251 4759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:27.328392 4759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "e86c78417b4246f3a01265f289734539"
format_stamp: "Formatted at 2026-06-19 14:55:27 on dist-test-slave-3qdh"
server_key: "f6cb72d1bb8cfbe09554d9f02aa0a2db"
server_key_iv: "251a6e145b845cd8d714ea24c35bee1c"
server_key_version: "encryptionkey@0"
I20260619 14:55:27.328493 4759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:27.353147 4759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:27.353682 4759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:27.353842 4759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:27.354310 4759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:55:27.355072 4759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:55:27.355122 4759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:27.355156 4759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:55:27.355208 4759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:27.362920 4759 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:34581
I20260619 14:55:27.362955 4886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:34581 every 8 connection(s)
I20260619 14:55:27.363651 4759 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:55:27.368127 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 4759
I20260619 14:55:27.368275 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:55:27.368621 19593 external_mini_cluster.cc:1428] Setting key dce158fb91a6d1cabf7ef3da008a88f1
I20260619 14:55:27.370441 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44735
--builtin_ntp_servers=127.19.34.84:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:55:27.372330 4738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:27.365162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:39337 (local address 127.19.34.126:44735)
0619 14:55:27.365283 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:27.365286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:27.366142 (+ 856us) server_negotiation.cc:407] Connection header received
0619 14:55:27.367395 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:27.367399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:27.367463 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:27.367565 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:27.368581 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.369269 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.370520 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.370871 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.370983 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:27.371811 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:27.371816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:27.371821 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:27.371836 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:27.371859 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:27.371870 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:27.371942 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:27.372045 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 14:55:27.372107 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:55:27.372980 4889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:27.365420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44735 (local address 127.19.34.65:39337)
0619 14:55:27.365978 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:27.366011 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:27.367190 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:27.367746 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:27.367754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:27.367841 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:27.368429 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:27.368443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.369417 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.369419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:27.370398 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:27.370407 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.370995 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.370998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:27.371118 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:27.371614 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:55:27.371616 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:27.371646 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:55:27.371695 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:27.371948 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:27.371955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:27.371958 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:55:27.372031 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 14:55:27.372224 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":104,"threads_started":1}
I20260619 14:55:27.374037 4887 heartbeater.cc:347] Connected to a master server at 127.19.34.126:44735
I20260619 14:55:27.374253 4887 heartbeater.cc:464] Registering TS with master...
I20260619 14:55:27.374765 4887 heartbeater.cc:511] Master 127.19.34.126:44735 requested a full tablet report, sending...
I20260619 14:55:27.376073 4699 ts_manager.cc:194] Registered new tserver with Master: e86c78417b4246f3a01265f289734539 (127.19.34.65:34581)
I20260619 14:55:27.377075 4699 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.34.65:39337
W20260619 14:55:27.468470 4890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:27.468647 4890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:27.468719 4890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:27.472275 4890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:27.472385 4890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:27.472476 4890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:55:27.476825 4890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:27.477607 4890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:27.478660 4890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:27.486553 4898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:27.486649 4896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:27.486728 4890 server_base.cc:1034] running on GCE node
W20260619 14:55:27.486891 4895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:27.487211 4890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:27.487679 4890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:27.488838 4890 hybrid_clock.cc:648] HybridClock initialized: now 1781880927488841 us; error 33 us; skew 500 ppm
I20260619 14:55:27.490275 4890 webserver.cc:460] Webserver started at http://127.19.34.66:40107/ using document root <none> and password file <none>
I20260619 14:55:27.490737 4890 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:27.490815 4890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:27.491001 4890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:27.492740 4890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "cb1f725c426f475391026d2929250588"
format_stamp: "Formatted at 2026-06-19 14:55:27 on dist-test-slave-3qdh"
server_key: "7b492dab195986bb09ce30c95df17196"
server_key_iv: "809540ee6914cece5b2b82cf8895750f"
server_key_version: "encryptionkey@0"
I20260619 14:55:27.493244 4890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "cb1f725c426f475391026d2929250588"
format_stamp: "Formatted at 2026-06-19 14:55:27 on dist-test-slave-3qdh"
server_key: "7b492dab195986bb09ce30c95df17196"
server_key_iv: "809540ee6914cece5b2b82cf8895750f"
server_key_version: "encryptionkey@0"
I20260619 14:55:27.496452 4890 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:55:27.498323 4904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:27.499588 4890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:27.499755 4890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "cb1f725c426f475391026d2929250588"
format_stamp: "Formatted at 2026-06-19 14:55:27 on dist-test-slave-3qdh"
server_key: "7b492dab195986bb09ce30c95df17196"
server_key_iv: "809540ee6914cece5b2b82cf8895750f"
server_key_version: "encryptionkey@0"
I20260619 14:55:27.499879 4890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:27.538087 4890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:27.538595 4890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:27.538770 4890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:27.539218 4890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:55:27.539944 4890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:55:27.539988 4890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:27.540046 4890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:55:27.540076 4890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:27.547760 4890 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:34989
I20260619 14:55:27.547787 5017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:34989 every 8 connection(s)
I20260619 14:55:27.548588 4890 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:55:27.556357 4738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:27.550051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:55209 (local address 127.19.34.126:44735)
0619 14:55:27.550190 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:27.550194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:27.551034 (+ 840us) server_negotiation.cc:407] Connection header received
0619 14:55:27.552201 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:27.552221 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:27.552279 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:27.552409 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:27.553194 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.553697 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.554679 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.554885 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.554986 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:27.555877 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:27.555880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:27.555882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:27.555911 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:27.555927 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:27.555935 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:27.556000 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:27.556137 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 14:55:27.556189 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:55:27.556546 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 4890
I20260619 14:55:27.556650 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:55:27.556922 19593 external_mini_cluster.cc:1428] Setting key 516307813373ac9123e41ae377db5bbc
I20260619 14:55:27.557125 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:27.550368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44735 (local address 127.19.34.66:55209)
0619 14:55:27.550916 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:27.550945 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:27.552023 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:27.552569 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:27.552575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:27.552657 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:27.553059 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:27.553070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.553809 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.553813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:27.554557 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:27.554564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.555012 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.555016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:27.555155 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:27.555658 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:55:27.555659 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:27.555691 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:55:27.555740 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:27.556011 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:27.556020 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:27.556030 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 14:55:27.556131 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 14:55:27.556404 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":107,"threads_started":1}
I20260619 14:55:27.558156 5018 heartbeater.cc:347] Connected to a master server at 127.19.34.126:44735
I20260619 14:55:27.558377 5018 heartbeater.cc:464] Registering TS with master...
I20260619 14:55:27.558884 5018 heartbeater.cc:511] Master 127.19.34.126:44735 requested a full tablet report, sending...
I20260619 14:55:27.559010 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44735
--builtin_ntp_servers=127.19.34.84:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:55:27.559787 4698 ts_manager.cc:194] Registered new tserver with Master: cb1f725c426f475391026d2929250588 (127.19.34.66:34989)
I20260619 14:55:27.560357 4698 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.34.66:55209
W20260619 14:55:27.654139 5021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:27.654352 5021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:27.654386 5021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:27.658013 5021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:27.658053 5021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:27.658099 5021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:55:27.662304 5021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:44735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:27.663102 5021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:27.664266 5021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:27.672173 5027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:27.672180 5026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:27.672370 5021 server_base.cc:1034] running on GCE node
W20260619 14:55:27.672180 5029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:27.672825 5021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:27.673343 5021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:27.674562 5021 hybrid_clock.cc:648] HybridClock initialized: now 1781880927674531 us; error 25 us; skew 500 ppm
I20260619 14:55:27.675992 5021 webserver.cc:460] Webserver started at http://127.19.34.67:42871/ using document root <none> and password file <none>
I20260619 14:55:27.676504 5021 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:27.676577 5021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:27.676760 5021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:27.678364 5021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "ef93acc905d541e3b09554e1eb27cc2a"
format_stamp: "Formatted at 2026-06-19 14:55:27 on dist-test-slave-3qdh"
server_key: "65b4021f76e02f42f7b145ce1ca39d4b"
server_key_iv: "7c11b7c62162175c3b1a6191247b305f"
server_key_version: "encryptionkey@0"
I20260619 14:55:27.678825 5021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "ef93acc905d541e3b09554e1eb27cc2a"
format_stamp: "Formatted at 2026-06-19 14:55:27 on dist-test-slave-3qdh"
server_key: "65b4021f76e02f42f7b145ce1ca39d4b"
server_key_iv: "7c11b7c62162175c3b1a6191247b305f"
server_key_version: "encryptionkey@0"
I20260619 14:55:27.681819 5021 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:55:27.683874 5035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:27.684954 5021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:27.685098 5021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "ef93acc905d541e3b09554e1eb27cc2a"
format_stamp: "Formatted at 2026-06-19 14:55:27 on dist-test-slave-3qdh"
server_key: "65b4021f76e02f42f7b145ce1ca39d4b"
server_key_iv: "7c11b7c62162175c3b1a6191247b305f"
server_key_version: "encryptionkey@0"
I20260619 14:55:27.685217 5021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:27.701097 5021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:27.701553 5021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:27.701705 5021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:27.702124 5021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:55:27.702912 5021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:55:27.702980 5021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:27.703045 5021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:55:27.703089 5021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:27.710574 5021 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:38521
I20260619 14:55:27.710608 5148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:38521 every 8 connection(s)
I20260619 14:55:27.711295 5021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:55:27.714665 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 5021
I20260619 14:55:27.714764 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:55:27.715023 19593 external_mini_cluster.cc:1428] Setting key 4f9e28355cca0568dd9b6fe43689b761
I20260619 14:55:27.719383 4738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:27.712875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:42489 (local address 127.19.34.126:44735)
0619 14:55:27.713036 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:27.713039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:27.713877 (+ 838us) server_negotiation.cc:407] Connection header received
0619 14:55:27.715180 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:27.715183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:27.715240 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:27.715321 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:27.716282 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.716849 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.717926 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.718178 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.718250 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:27.719018 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:27.719020 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:27.719022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:27.719031 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:27.719047 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:27.719054 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:27.719107 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:27.719249 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 14:55:27.719290 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 14:55:27.720055 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:27.713202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:44735 (local address 127.19.34.67:42489)
0619 14:55:27.713748 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:27.713780 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:27.714955 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:27.715480 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:27.715487 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:27.715576 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:27.716129 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:27.716149 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.716976 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.716979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:27.717827 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:27.717836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.718286 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.718288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:27.718399 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:27.718821 (+ 422us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:55:27.718822 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:27.718853 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:55:27.718903 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:27.719120 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:27.719125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:27.719128 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:55:27.719238 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 14:55:27.719439 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":104,"threads_started":1}
I20260619 14:55:27.720989 5149 heartbeater.cc:347] Connected to a master server at 127.19.34.126:44735
I20260619 14:55:27.721171 5149 heartbeater.cc:464] Registering TS with master...
I20260619 14:55:27.721537 5149 heartbeater.cc:511] Master 127.19.34.126:44735 requested a full tablet report, sending...
I20260619 14:55:27.722384 4698 ts_manager.cc:194] Registered new tserver with Master: ef93acc905d541e3b09554e1eb27cc2a (127.19.34.67:38521)
I20260619 14:55:27.722916 4698 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.34.67:42489
I20260619 14:55:27.728595 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:55:27.733605 4738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:27.730247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35896 (local address 127.19.34.126:44735)
0619 14:55:27.730342 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:27.730345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:27.730506 (+ 161us) server_negotiation.cc:407] Connection header received
0619 14:55:27.730575 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:27.730577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:27.730612 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:27.730666 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:27.730852 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.731340 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.732053 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.732287 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.732429 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:27.733212 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:27.733215 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:27.733217 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:27.733228 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:27.733247 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:27.733260 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:27.733335 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:27.733435 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 14:55:27.733488 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:55:27.735764 4698 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:55:27.738322 4698 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:55:27.749853 5162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:27.745136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:38521 (local address 127.0.0.1:41350)
0619 14:55:27.745585 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:27.745634 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:27.745745 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:27.746241 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:27.746246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:27.746263 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:27.746374 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:27.746380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.747498 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.747501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:27.748487 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:27.748495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.749451 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.749454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:27.749552 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:27.749585 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 14:55:27.749653 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 14:55:27.749743 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":71,"threads_started":1}
I20260619 14:55:27.750590 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:27.745201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41350 (local address 127.19.34.67:38521)
0619 14:55:27.745819 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:27.745829 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:27.745848 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:55:27.745926 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:27.745931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:27.746108 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:27.746242 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:27.746492 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.747405 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.748617 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.749333 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.750166 (+ 833us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:27.750317 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:27.750359 (+ 42us) server_negotiation.cc:299] Negotiation successful
0619 14:55:27.750455 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":125,"threads_started":1}
I20260619 14:55:27.750749 5160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:27.745811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:34989 (local address 127.0.0.1:38072)
0619 14:55:27.746437 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:27.746449 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:27.746511 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:27.747319 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:27.747322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:27.747339 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:27.747427 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:27.747435 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.748559 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.748564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:27.749366 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:27.749375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.750248 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.750251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:27.750398 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:27.750448 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 14:55:27.750532 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 14:55:27.750595 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":39,"thread_start_us":85,"threads_started":1}
I20260619 14:55:27.751019 5164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:27.745239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34581 (local address 127.0.0.1:44446)
0619 14:55:27.746010 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:27.746021 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:27.746090 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:27.746610 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:27.746612 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:27.746626 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:27.746697 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:27.746702 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.747996 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.747999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:27.749075 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:27.749085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.750223 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.750226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:27.750322 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:27.750437 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 14:55:27.750502 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 14:55:27.750562 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":55,"thread_start_us":98,"threads_started":1}
I20260619 14:55:27.751166 5165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:27.746180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38072 (local address 127.19.34.66:34989)
0619 14:55:27.746648 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:27.746659 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:27.746678 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:55:27.746932 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:27.746936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:27.747163 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:27.747322 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:27.747559 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.748455 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.749483 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.750135 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.750236 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:27.750414 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:27.750940 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 14:55:27.751022 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":97,"threads_started":1}
I20260619 14:55:27.751394 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:27.745671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44446 (local address 127.19.34.65:34581)
0619 14:55:27.746071 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:27.746082 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:27.746106 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:55:27.746175 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:27.746180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:27.746345 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:27.746480 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:27.746856 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.747882 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.749204 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:27.749921 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:27.750011 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:27.750170 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:27.750552 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 14:55:27.750642 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":128,"threads_started":1}
I20260619 14:55:27.753208 5083 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6ff8ca95184b7089130d7c71221968 (DEFAULT_TABLE table=test-table [id=1b2e226d0db54045b304b39498c04d27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:55:27.753459 4952 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6ff8ca95184b7089130d7c71221968 (DEFAULT_TABLE table=test-table [id=1b2e226d0db54045b304b39498c04d27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:55:27.753774 4821 tablet_service.cc:1467] Processing CreateTablet for tablet 3f6ff8ca95184b7089130d7c71221968 (DEFAULT_TABLE table=test-table [id=1b2e226d0db54045b304b39498c04d27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:55:27.754113 4952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6ff8ca95184b7089130d7c71221968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:27.754127 5083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6ff8ca95184b7089130d7c71221968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:27.754516 4821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f6ff8ca95184b7089130d7c71221968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:27.759245 5166 tablet_bootstrap.cc:492] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588: Bootstrap starting.
I20260619 14:55:27.760587 5167 tablet_bootstrap.cc:492] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a: Bootstrap starting.
I20260619 14:55:27.760952 5168 tablet_bootstrap.cc:492] T 3f6ff8ca95184b7089130d7c71221968 P e86c78417b4246f3a01265f289734539: Bootstrap starting.
I20260619 14:55:27.761668 5166 tablet_bootstrap.cc:654] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:27.762466 5167 tablet_bootstrap.cc:654] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:27.762477 5166 log.cc:824] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:27.762506 5168 tablet_bootstrap.cc:654] T 3f6ff8ca95184b7089130d7c71221968 P e86c78417b4246f3a01265f289734539: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:27.763166 5167 log.cc:824] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:27.763295 5168 log.cc:824] T 3f6ff8ca95184b7089130d7c71221968 P e86c78417b4246f3a01265f289734539: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:27.764421 5166 tablet_bootstrap.cc:492] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588: No bootstrap required, opened a new log
I20260619 14:55:27.764585 5166 ts_tablet_manager.cc:1397] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:55:27.764989 5167 tablet_bootstrap.cc:492] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a: No bootstrap required, opened a new log
I20260619 14:55:27.765158 5167 ts_tablet_manager.cc:1397] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:55:27.765482 5168 tablet_bootstrap.cc:492] T 3f6ff8ca95184b7089130d7c71221968 P e86c78417b4246f3a01265f289734539: No bootstrap required, opened a new log
I20260619 14:55:27.765659 5168 ts_tablet_manager.cc:1397] T 3f6ff8ca95184b7089130d7c71221968 P e86c78417b4246f3a01265f289734539: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:55:27.768999 5168 raft_consensus.cc:348] T 3f6ff8ca95184b7089130d7c71221968 P e86c78417b4246f3a01265f289734539 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef93acc905d541e3b09554e1eb27cc2a" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38521 } } peers { permanent_uuid: "e86c78417b4246f3a01265f289734539" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34581 } } peers { permanent_uuid: "cb1f725c426f475391026d2929250588" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34989 } }
I20260619 14:55:27.769209 5168 raft_consensus.cc:374] T 3f6ff8ca95184b7089130d7c71221968 P e86c78417b4246f3a01265f289734539 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:27.769260 5168 raft_consensus.cc:729] T 3f6ff8ca95184b7089130d7c71221968 P e86c78417b4246f3a01265f289734539 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e86c78417b4246f3a01265f289734539, State: Initialized, Role: FOLLOWER
I20260619 14:55:27.769217 5167 raft_consensus.cc:348] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef93acc905d541e3b09554e1eb27cc2a" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38521 } } peers { permanent_uuid: "e86c78417b4246f3a01265f289734539" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34581 } } peers { permanent_uuid: "cb1f725c426f475391026d2929250588" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34989 } }
I20260619 14:55:27.769454 5167 raft_consensus.cc:374] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:27.769516 5167 raft_consensus.cc:729] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef93acc905d541e3b09554e1eb27cc2a, State: Initialized, Role: FOLLOWER
I20260619 14:55:27.769368 5166 raft_consensus.cc:348] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef93acc905d541e3b09554e1eb27cc2a" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38521 } } peers { permanent_uuid: "e86c78417b4246f3a01265f289734539" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34581 } } peers { permanent_uuid: "cb1f725c426f475391026d2929250588" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34989 } }
I20260619 14:55:27.769578 5166 raft_consensus.cc:374] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:27.769625 5166 raft_consensus.cc:729] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb1f725c426f475391026d2929250588, State: Initialized, Role: FOLLOWER
I20260619 14:55:27.769740 5168 consensus_queue.cc:260] T 3f6ff8ca95184b7089130d7c71221968 P e86c78417b4246f3a01265f289734539 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef93acc905d541e3b09554e1eb27cc2a" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38521 } } peers { permanent_uuid: "e86c78417b4246f3a01265f289734539" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34581 } } peers { permanent_uuid: "cb1f725c426f475391026d2929250588" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34989 } }
I20260619 14:55:27.769946 5166 consensus_queue.cc:260] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef93acc905d541e3b09554e1eb27cc2a" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38521 } } peers { permanent_uuid: "e86c78417b4246f3a01265f289734539" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34581 } } peers { permanent_uuid: "cb1f725c426f475391026d2929250588" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34989 } }
I20260619 14:55:27.769946 5167 consensus_queue.cc:260] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef93acc905d541e3b09554e1eb27cc2a" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38521 } } peers { permanent_uuid: "e86c78417b4246f3a01265f289734539" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34581 } } peers { permanent_uuid: "cb1f725c426f475391026d2929250588" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34989 } }
I20260619 14:55:27.770390 4887 heartbeater.cc:503] Master 127.19.34.126:44735 was elected leader, sending a full tablet report...
I20260619 14:55:27.770627 5168 ts_tablet_manager.cc:1428] T 3f6ff8ca95184b7089130d7c71221968 P e86c78417b4246f3a01265f289734539: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:55:27.770735 5166 ts_tablet_manager.cc:1428] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:55:27.770987 5018 heartbeater.cc:503] Master 127.19.34.126:44735 was elected leader, sending a full tablet report...
I20260619 14:55:27.771525 5167 ts_tablet_manager.cc:1428] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:55:27.771981 5149 heartbeater.cc:503] Master 127.19.34.126:44735 was elected leader, sending a full tablet report...
W20260619 14:55:27.799328 5019 tablet.cc:2369] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:55:27.864687 4888 tablet.cc:2369] T 3f6ff8ca95184b7089130d7c71221968 P e86c78417b4246f3a01265f289734539: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:55:27.962133 5150 tablet.cc:2369] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:55:28.001639 5174 raft_consensus.cc:482] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:55:28.001861 5174 raft_consensus.cc:504] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef93acc905d541e3b09554e1eb27cc2a" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38521 } } peers { permanent_uuid: "e86c78417b4246f3a01265f289734539" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34581 } } peers { permanent_uuid: "cb1f725c426f475391026d2929250588" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34989 } }
I20260619 14:55:28.002971 5174 leader_election.cc:288] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef93acc905d541e3b09554e1eb27cc2a (127.19.34.67:38521), e86c78417b4246f3a01265f289734539 (127.19.34.65:34581)
I20260619 14:55:28.006235 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:28.003296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:48095 (local address 127.19.34.67:38521)
0619 14:55:28.003434 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:28.003450 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:28.003464 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:55:28.003532 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:28.003534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:28.003622 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:28.003721 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:28.003968 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:28.004469 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:28.005413 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:28.005764 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:28.005877 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:28.005924 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:28.006074 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 14:55:28.006134 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:55:28.006235 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:28.003161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:38521 (local address 127.19.34.66:48095)
0619 14:55:28.003294 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:28.003310 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:28.003413 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:28.003730 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:28.003733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:28.003748 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:28.003855 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:28.003860 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:28.004567 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:28.004570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:28.005297 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:28.005305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:28.005869 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:28.005872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:28.005957 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:28.005976 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:55:28.006058 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 14:55:28.006120 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 14:55:28.006369 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:28.003470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:33247 (local address 127.19.34.65:34581)
0619 14:55:28.003609 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:28.003614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:28.003666 (+ 52us) server_negotiation.cc:407] Connection header received
0619 14:55:28.003797 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:28.003805 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:28.003843 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:28.003930 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:28.004164 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:28.004638 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:28.005555 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:28.005885 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:28.005993 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:28.006032 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:28.006184 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 14:55:28.006240 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:55:28.006402 5175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:28.003252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34581 (local address 127.19.34.66:33247)
0619 14:55:28.003609 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:28.003621 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:28.003703 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:28.003951 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:28.003953 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:28.003966 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:28.004068 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:28.004072 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:28.004736 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:28.004739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:28.005451 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:28.005459 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:28.005989 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:28.005992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:28.006077 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:28.006098 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:55:28.006171 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 14:55:28.006227 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260619 14:55:28.006881 5103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6ff8ca95184b7089130d7c71221968" candidate_uuid: "cb1f725c426f475391026d2929250588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef93acc905d541e3b09554e1eb27cc2a" is_pre_election: true
I20260619 14:55:28.006927 4841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6ff8ca95184b7089130d7c71221968" candidate_uuid: "cb1f725c426f475391026d2929250588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e86c78417b4246f3a01265f289734539" is_pre_election: true
I20260619 14:55:28.007141 5103 raft_consensus.cc:2454] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb1f725c426f475391026d2929250588 in term 0.
I20260619 14:55:28.007229 4841 raft_consensus.cc:2454] T 3f6ff8ca95184b7089130d7c71221968 P e86c78417b4246f3a01265f289734539 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb1f725c426f475391026d2929250588 in term 0.
I20260619 14:55:28.007604 4907 leader_election.cc:302] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb1f725c426f475391026d2929250588, ef93acc905d541e3b09554e1eb27cc2a; no voters:
I20260619 14:55:28.007853 5174 raft_consensus.cc:2781] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:55:28.007927 5174 raft_consensus.cc:482] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:55:28.007956 5174 raft_consensus.cc:3037] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:28.008916 5174 raft_consensus.cc:504] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef93acc905d541e3b09554e1eb27cc2a" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38521 } } peers { permanent_uuid: "e86c78417b4246f3a01265f289734539" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34581 } } peers { permanent_uuid: "cb1f725c426f475391026d2929250588" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34989 } }
I20260619 14:55:28.009229 5174 leader_election.cc:288] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 [CANDIDATE]: Term 1 election: Requested vote from peers ef93acc905d541e3b09554e1eb27cc2a (127.19.34.67:38521), e86c78417b4246f3a01265f289734539 (127.19.34.65:34581)
I20260619 14:55:28.009513 5103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6ff8ca95184b7089130d7c71221968" candidate_uuid: "cb1f725c426f475391026d2929250588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef93acc905d541e3b09554e1eb27cc2a"
I20260619 14:55:28.009550 4841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f6ff8ca95184b7089130d7c71221968" candidate_uuid: "cb1f725c426f475391026d2929250588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e86c78417b4246f3a01265f289734539"
I20260619 14:55:28.009608 5103 raft_consensus.cc:3037] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:28.009675 4841 raft_consensus.cc:3037] T 3f6ff8ca95184b7089130d7c71221968 P e86c78417b4246f3a01265f289734539 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:28.010538 5103 raft_consensus.cc:2454] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb1f725c426f475391026d2929250588 in term 1.
I20260619 14:55:28.010627 4841 raft_consensus.cc:2454] T 3f6ff8ca95184b7089130d7c71221968 P e86c78417b4246f3a01265f289734539 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb1f725c426f475391026d2929250588 in term 1.
I20260619 14:55:28.010811 4907 leader_election.cc:302] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb1f725c426f475391026d2929250588, ef93acc905d541e3b09554e1eb27cc2a; no voters:
I20260619 14:55:28.010998 5174 raft_consensus.cc:2781] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:28.011193 5174 raft_consensus.cc:686] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 [term 1 LEADER]: Becoming Leader. State: Replica: cb1f725c426f475391026d2929250588, State: Running, Role: LEADER
I20260619 14:55:28.011468 5174 consensus_queue.cc:237] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef93acc905d541e3b09554e1eb27cc2a" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38521 } } peers { permanent_uuid: "e86c78417b4246f3a01265f289734539" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34581 } } peers { permanent_uuid: "cb1f725c426f475391026d2929250588" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34989 } }
I20260619 14:55:28.014339 4698 catalog_manager.cc:5495] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb1f725c426f475391026d2929250588 (127.19.34.66). New cstate: current_term: 1 leader_uuid: "cb1f725c426f475391026d2929250588" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef93acc905d541e3b09554e1eb27cc2a" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e86c78417b4246f3a01265f289734539" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb1f725c426f475391026d2929250588" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 34989 } health_report { overall_health: HEALTHY } } }
I20260619 14:55:28.037014 5165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:28.033861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38088 (local address 127.19.34.66:34989)
0619 14:55:28.033986 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:28.033989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:28.034002 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:55:28.034124 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:28.034126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:28.034172 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:28.034253 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:28.034424 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:28.034878 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:28.035705 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:28.035930 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:28.036011 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:28.036238 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:28.036666 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:28.036753 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:28.036808 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 14:55:28.036847 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:55:28.042167 5103 raft_consensus.cc:1261] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a [term 1 FOLLOWER]: Refusing update from remote peer cb1f725c426f475391026d2929250588: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:55:28.042156 4841 raft_consensus.cc:1261] T 3f6ff8ca95184b7089130d7c71221968 P e86c78417b4246f3a01265f289734539 [term 1 FOLLOWER]: Refusing update from remote peer cb1f725c426f475391026d2929250588: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:55:28.042900 5176 consensus_queue.cc:1035] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef93acc905d541e3b09554e1eb27cc2a" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 38521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:55:28.043076 5174 consensus_queue.cc:1035] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e86c78417b4246f3a01265f289734539" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:55:28.056432 4698 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:55:28.056566 4698 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:55:28.058713 4698 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 471d8f7c2ddb4616841fb0769f62ff3c: Sending DeleteTablet for 3 replicas of tablet 3f6ff8ca95184b7089130d7c71221968
I20260619 14:55:28.059358 4821 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6ff8ca95184b7089130d7c71221968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:55:28 UTC) from {username='slave'} at 127.0.0.1:44446
I20260619 14:55:28.059398 5083 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6ff8ca95184b7089130d7c71221968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:55:28 UTC) from {username='slave'} at 127.0.0.1:41350
I20260619 14:55:28.059427 4952 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f6ff8ca95184b7089130d7c71221968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:55:28 UTC) from {username='slave'} at 127.0.0.1:38072
I20260619 14:55:28.059775 5189 tablet_replica.cc:330] T 3f6ff8ca95184b7089130d7c71221968 P e86c78417b4246f3a01265f289734539: stopping tablet replica
I20260619 14:55:28.060060 5189 raft_consensus.cc:2229] T 3f6ff8ca95184b7089130d7c71221968 P e86c78417b4246f3a01265f289734539 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:55:28.060187 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 4759
I20260619 14:55:28.064460 5191 tablet_replica.cc:330] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a: stopping tablet replica
I20260619 14:55:28.064708 5190 tablet_replica.cc:330] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588: stopping tablet replica
I20260619 14:55:28.064721 5191 raft_consensus.cc:2229] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:55:28.064944 5191 raft_consensus.cc:2258] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:55:28.065052 5190 raft_consensus.cc:2229] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:55:28.065419 5190 raft_consensus.cc:2258] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:55:28.066334 5191 ts_tablet_manager.cc:1905] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:55:28.066694 5190 ts_tablet_manager.cc:1905] T 3f6ff8ca95184b7089130d7c71221968 P cb1f725c426f475391026d2929250588: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 14:55:28.067339 4683 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.34.65:34581 (error 108)
I20260619 14:55:28.067581 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 4890
I20260619 14:55:28.069818 5191 ts_tablet_manager.cc:1918] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:55:28.069904 5191 log.cc:1196] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/3f6ff8ca95184b7089130d7c71221968
I20260619 14:55:28.069933 5160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:28.069639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34581 (local address 127.0.0.1:44456)
0619 14:55:28.069788 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:28.069853 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.65:34581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 14:55:28.070109 4683 catalog_manager.cc:4584] TS e86c78417b4246f3a01265f289734539 (127.19.34.65:34581): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3f6ff8ca95184b7089130d7c71221968: Network error: Client connection negotiation failed: client connection to 127.19.34.65:34581: connect: Connection refused (error 111)
I20260619 14:55:28.070232 5191 ts_tablet_manager.cc:1939] T 3f6ff8ca95184b7089130d7c71221968 P ef93acc905d541e3b09554e1eb27cc2a: Deleting consensus metadata
I20260619 14:55:28.071131 4685 catalog_manager.cc:4841] TS ef93acc905d541e3b09554e1eb27cc2a (127.19.34.67:38521): tablet 3f6ff8ca95184b7089130d7c71221968 (table test-table [id=1b2e226d0db54045b304b39498c04d27]) successfully deleted
I20260619 14:55:28.073572 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 5021
I20260619 14:55:28.074353 5160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:28.073647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:34989 (local address 127.0.0.1:38104)
0619 14:55:28.074217 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:28.074275 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.66:34989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488}
I20260619 14:55:28.079770 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 4668
2026-06-19T14:55:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1110 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:28 dist-test-slave-3qdh krb5kdc[5195](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:28 dist-test-slave-3qdh krb5kdc[5195](info): set up 2 sockets
Jun 19 14:55:28 dist-test-slave-3qdh krb5kdc[5195](info): commencing operation
krb5kdc: starting...
W20260619 14:55:29.039564 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:29 dist-test-slave-3qdh krb5kdc[5195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:55:29.145639 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:42829
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:42829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:29.242468 5208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:29.242745 5208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:29.242820 5208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:29.246658 5208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:29.246757 5208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:29.246795 5208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:29.246851 5208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:29.246888 5208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:29.251672 5208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:42829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:42829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:29.252574 5208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:29.253449 5208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:29.260475 5216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:29.260563 5214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:29.260475 5213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:29.260700 5208 server_base.cc:1034] running on GCE node
I20260619 14:55:29.261189 5208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:29.262105 5208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:29.263368 5208 hybrid_clock.cc:648] HybridClock initialized: now 1781880929263357 us; error 30 us; skew 500 ppm
Jun 19 14:55:29 dist-test-slave-3qdh krb5kdc[5195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880929, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:29.265678 5208 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:55:29.266368 5208 webserver.cc:460] Webserver started at http://127.19.34.126:42929/ using document root <none> and password file <none>
I20260619 14:55:29.266811 5208 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:29.266893 5208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:29.267076 5208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:29.268838 5208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "2423ece1751e4a3ab8c03f71bd667cd0"
format_stamp: "Formatted at 2026-06-19 14:55:29 on dist-test-slave-3qdh"
server_key: "9ea3ec2961e840cd49d90085bb4c715c"
server_key_iv: "39218ae47407f1e900c6c11c2b93a2be"
server_key_version: "encryptionkey@0"
I20260619 14:55:29.269361 5208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "2423ece1751e4a3ab8c03f71bd667cd0"
format_stamp: "Formatted at 2026-06-19 14:55:29 on dist-test-slave-3qdh"
server_key: "9ea3ec2961e840cd49d90085bb4c715c"
server_key_iv: "39218ae47407f1e900c6c11c2b93a2be"
server_key_version: "encryptionkey@0"
I20260619 14:55:29.272281 5208 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:55:29.274498 5223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:29.275496 5208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:55:29.275643 5208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "2423ece1751e4a3ab8c03f71bd667cd0"
format_stamp: "Formatted at 2026-06-19 14:55:29 on dist-test-slave-3qdh"
server_key: "9ea3ec2961e840cd49d90085bb4c715c"
server_key_iv: "39218ae47407f1e900c6c11c2b93a2be"
server_key_version: "encryptionkey@0"
I20260619 14:55:29.275745 5208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:29.286540 5208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:29.289090 5208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:29.289310 5208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:29.296119 5208 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:42829
I20260619 14:55:29.296181 5275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:42829 every 8 connection(s)
I20260619 14:55:29.296978 5208 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:29.299228 5276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:29.300192 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 5208
I20260619 14:55:29.300359 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:29.300654 19593 external_mini_cluster.cc:1428] Setting key b489c6034bc26ae763f32aaf91665b76
I20260619 14:55:29.305240 5276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0: Bootstrap starting.
I20260619 14:55:29.307381 5276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:29.308127 5276 log.cc:824] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:55:29 dist-test-slave-3qdh krb5kdc[5195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:55:29.310014 5276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0: No bootstrap required, opened a new log
I20260619 14:55:29.312995 5276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2423ece1751e4a3ab8c03f71bd667cd0" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42829 } }
I20260619 14:55:29.313302 5276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:29.313397 5276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2423ece1751e4a3ab8c03f71bd667cd0, State: Initialized, Role: FOLLOWER
I20260619 14:55:29.313831 5276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2423ece1751e4a3ab8c03f71bd667cd0" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42829 } }
I20260619 14:55:29.313961 5276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:29.314038 5276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:29.314047 5279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:29.301879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57792 (local address 127.19.34.126:42829)
0619 14:55:29.302254 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:29.302272 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:29.302308 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:55:29.303070 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:29.303089 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:29.303445 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:29.303783 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:29.304494 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.305806 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.306568 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.307130 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.307306 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:29.309708 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:29.309714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:29.309726 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:29.309752 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:29.311272 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:29.311680 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:29.311687 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:29.311691 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:29.311795 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:29.312057 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:29.312061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:29.312063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:29.312473 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:29.312681 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:29.312940 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 14:55:29.313193 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":93,"threads_started":1}
I20260619 14:55:29.314132 5276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:29.314999 5276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2423ece1751e4a3ab8c03f71bd667cd0" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42829 } }
I20260619 14:55:29.315303 5276 leader_election.cc:302] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2423ece1751e4a3ab8c03f71bd667cd0; no voters:
I20260619 14:55:29.315562 5276 leader_election.cc:288] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:29.315649 5281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:29.315848 5281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0 [term 1 LEADER]: Becoming Leader. State: Replica: 2423ece1751e4a3ab8c03f71bd667cd0, State: Running, Role: LEADER
I20260619 14:55:29.316071 5281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2423ece1751e4a3ab8c03f71bd667cd0" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42829 } }
I20260619 14:55:29.316673 5276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:29.317184 5282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2423ece1751e4a3ab8c03f71bd667cd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2423ece1751e4a3ab8c03f71bd667cd0" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42829 } } }
I20260619 14:55:29.317234 5283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2423ece1751e4a3ab8c03f71bd667cd0. Latest consensus state: current_term: 1 leader_uuid: "2423ece1751e4a3ab8c03f71bd667cd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2423ece1751e4a3ab8c03f71bd667cd0" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42829 } } }
I20260619 14:55:29.317517 5283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:29.317512 5282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:29.317898 5290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:29.320164 5290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:29.324872 5290 catalog_manager.cc:1347] Generated new cluster ID: 65a287937e56409aa2dc61551c286473
I20260619 14:55:29.324944 5290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:29.340816 5290 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:29.341503 5290 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:29.348169 5290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0: Generated new TSK 0
I20260619 14:55:29.348764 5290 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.34.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
I20260619 14:55:29.381163 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:42829
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:29.477432 5302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:29.477643 5302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:29.477680 5302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:29.481526 5302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:29.481575 5302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:29.481673 5302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:55:29.486421 5302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:42829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:29.487161 5302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:29.488088 5302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:29.496112 5308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:29.496156 5310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:29.496196 5302 server_base.cc:1034] running on GCE node
W20260619 14:55:29.496124 5307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:29.496819 5302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:29.497387 5302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:29.498546 5302 hybrid_clock.cc:648] HybridClock initialized: now 1781880929498516 us; error 39 us; skew 500 ppm
Jun 19 14:55:29 dist-test-slave-3qdh krb5kdc[5195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880929, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:29.501099 5302 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:55:29.501813 5302 webserver.cc:460] Webserver started at http://127.19.34.65:41851/ using document root <none> and password file <none>
I20260619 14:55:29.502300 5302 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:29.502363 5302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:29.502646 5302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:29.504484 5302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "c72b05af8218482ab52f9f412208a040"
format_stamp: "Formatted at 2026-06-19 14:55:29 on dist-test-slave-3qdh"
server_key: "fffd5e2e882ebae57ecd359cf14d2102"
server_key_iv: "794310aba765a46f81d976d90a644c60"
server_key_version: "encryptionkey@0"
I20260619 14:55:29.505044 5302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "c72b05af8218482ab52f9f412208a040"
format_stamp: "Formatted at 2026-06-19 14:55:29 on dist-test-slave-3qdh"
server_key: "fffd5e2e882ebae57ecd359cf14d2102"
server_key_iv: "794310aba765a46f81d976d90a644c60"
server_key_version: "encryptionkey@0"
I20260619 14:55:29.508234 5302 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:55:29.510303 5317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:29.511292 5302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:55:29.511430 5302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "c72b05af8218482ab52f9f412208a040"
format_stamp: "Formatted at 2026-06-19 14:55:29 on dist-test-slave-3qdh"
server_key: "fffd5e2e882ebae57ecd359cf14d2102"
server_key_iv: "794310aba765a46f81d976d90a644c60"
server_key_version: "encryptionkey@0"
I20260619 14:55:29.511529 5302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:29.534266 5302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:29.536566 5302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:29.536731 5302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:29.537163 5302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:55:29.537858 5302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:55:29.537928 5302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:29.537984 5302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:55:29.538031 5302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:29.546495 5302 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:42289
I20260619 14:55:29.546530 5430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:42289 every 8 connection(s)
I20260619 14:55:29.547188 5302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
Jun 19 14:55:29 dist-test-slave-3qdh krb5kdc[5195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880929, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:55:29.556111 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 5302
I20260619 14:55:29.556255 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:55:29.556607 19593 external_mini_cluster.cc:1428] Setting key d5d77404a20490cf54e71fb6db670b28
I20260619 14:55:29.559049 5279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:29.548674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:51411 (local address 127.19.34.126:42829)
0619 14:55:29.548786 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:29.548791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:29.549403 (+ 612us) server_negotiation.cc:407] Connection header received
0619 14:55:29.550477 (+ 1074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:29.550480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:29.550531 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:29.550599 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:29.551517 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.552000 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.553226 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.553452 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.553581 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:29.555659 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:29.555663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:29.555665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:29.555692 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:29.557096 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:29.557615 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:29.557620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:29.557621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:29.557733 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:29.558022 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:29.558025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:29.558027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:29.558246 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:29.558349 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:29.558710 (+ 361us) server_negotiation.cc:299] Negotiation successful
0619 14:55:29.558836 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:55:29.559547 5433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:29.548894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:42829 (local address 127.19.34.65:51411)
0619 14:55:29.549280 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:29.549312 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:29.550299 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:29.550758 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:29.550763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:29.551029 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:29.551389 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:29.551399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.552121 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.552124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:29.553051 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:29.553061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.553608 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.553611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:29.553770 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:29.554281 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:29.554295 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:29.555505 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:29.557227 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:29.557231 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:29.557239 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:29.557462 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:29.557814 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:29.557817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:29.557818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:29.557929 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:29.558345 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:29.558348 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:29.558509 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 14:55:29.558626 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 14:55:29.558814 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":105,"threads_started":1}
I20260619 14:55:29.560705 5431 heartbeater.cc:347] Connected to a master server at 127.19.34.126:42829
I20260619 14:55:29.560889 5431 heartbeater.cc:464] Registering TS with master...
I20260619 14:55:29.561302 5431 heartbeater.cc:511] Master 127.19.34.126:42829 requested a full tablet report, sending...
I20260619 14:55:29.562687 5240 ts_manager.cc:194] Registered new tserver with Master: c72b05af8218482ab52f9f412208a040 (127.19.34.65:42289)
I20260619 14:55:29.563771 5240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:51411
WARNING: no policy specified for kudu/127.19.34.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
I20260619 14:55:29.580760 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:42829
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:29.675800 5436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:29.675994 5436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:29.676029 5436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:29.679778 5436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:29.679816 5436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:29.679899 5436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:55:29.684412 5436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:42829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:29.685168 5436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:29.685968 5436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:29.693981 5444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:29.694011 5442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:29.693993 5441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:29.694253 5436 server_base.cc:1034] running on GCE node
I20260619 14:55:29.694638 5436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:29.695191 5436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:29.696396 5436 hybrid_clock.cc:648] HybridClock initialized: now 1781880929696385 us; error 23 us; skew 500 ppm
Jun 19 14:55:29 dist-test-slave-3qdh krb5kdc[5195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880929, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:29.698892 5436 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:55:29.699591 5436 webserver.cc:460] Webserver started at http://127.19.34.66:34129/ using document root <none> and password file <none>
I20260619 14:55:29.700054 5436 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:29.700138 5436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:29.700407 5436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:29.702123 5436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "4e429d417c6b45568ca6910d9938c2e9"
format_stamp: "Formatted at 2026-06-19 14:55:29 on dist-test-slave-3qdh"
server_key: "c98a07b988fa51319d470336abafb8bc"
server_key_iv: "58653ba361dcb11e5540eb7122608e15"
server_key_version: "encryptionkey@0"
I20260619 14:55:29.702672 5436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "4e429d417c6b45568ca6910d9938c2e9"
format_stamp: "Formatted at 2026-06-19 14:55:29 on dist-test-slave-3qdh"
server_key: "c98a07b988fa51319d470336abafb8bc"
server_key_iv: "58653ba361dcb11e5540eb7122608e15"
server_key_version: "encryptionkey@0"
I20260619 14:55:29.705830 5436 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:55:29.707803 5451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:29.708976 5436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:29.709120 5436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "4e429d417c6b45568ca6910d9938c2e9"
format_stamp: "Formatted at 2026-06-19 14:55:29 on dist-test-slave-3qdh"
server_key: "c98a07b988fa51319d470336abafb8bc"
server_key_iv: "58653ba361dcb11e5540eb7122608e15"
server_key_version: "encryptionkey@0"
I20260619 14:55:29.709246 5436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:29.724429 5436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:29.726820 5436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:29.726996 5436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:29.727437 5436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:55:29.728132 5436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:55:29.728212 5436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:29.728273 5436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:55:29.728332 5436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:29.736224 5436 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:39137
I20260619 14:55:29.736248 5564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:39137 every 8 connection(s)
I20260619 14:55:29.737052 5436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
Jun 19 14:55:29 dist-test-slave-3qdh krb5kdc[5195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880929, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:55:29.745366 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 5436
I20260619 14:55:29.745478 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:55:29.745780 19593 external_mini_cluster.cc:1428] Setting key e3a02d93a2d07b1bb76d291c81859296
I20260619 14:55:29.749069 5279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:29.738631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:38943 (local address 127.19.34.126:42829)
0619 14:55:29.738767 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:29.738770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:29.739350 (+ 580us) server_negotiation.cc:407] Connection header received
0619 14:55:29.740394 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:29.740397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:29.740444 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:29.740508 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:29.741428 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.741890 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.742964 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.743172 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.743246 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:29.745535 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:29.745538 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:29.745541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:29.745565 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:29.746918 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:29.747452 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:29.747456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:29.747458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:29.747556 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:29.747882 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:29.747888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:29.747893 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:29.748132 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:29.748247 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:29.748710 (+ 463us) server_negotiation.cc:299] Negotiation successful
0619 14:55:29.748829 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:55:29.749816 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:29.738834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:42829 (local address 127.19.34.66:38943)
0619 14:55:29.739209 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:29.739243 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:29.740203 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:29.740643 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:29.740649 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:29.740912 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:29.741294 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:29.741322 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.742007 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.742010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:29.742855 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:29.742865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.743289 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.743292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:29.743415 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:29.743877 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:29.743890 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:29.745355 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:29.747027 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:29.747032 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:29.747042 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:29.747343 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:29.747646 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:29.747648 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:29.747649 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:29.747759 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:29.748244 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:29.748248 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:29.748479 (+ 231us) client_negotiation.cc:769] Sending connection context
0619 14:55:29.748618 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 14:55:29.748879 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":94,"threads_started":1}
I20260619 14:55:29.750828 5565 heartbeater.cc:347] Connected to a master server at 127.19.34.126:42829
I20260619 14:55:29.751024 5565 heartbeater.cc:464] Registering TS with master...
I20260619 14:55:29.751502 5565 heartbeater.cc:511] Master 127.19.34.126:42829 requested a full tablet report, sending...
I20260619 14:55:29.752461 5240 ts_manager.cc:194] Registered new tserver with Master: 4e429d417c6b45568ca6910d9938c2e9 (127.19.34.66:39137)
I20260619 14:55:29.752991 5240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:38943
WARNING: no policy specified for kudu/127.19.34.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
I20260619 14:55:29.771904 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:42829
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:40287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:29.867528 5570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:29.867712 5570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:29.867780 5570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:29.871301 5570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:29.871343 5570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:29.871434 5570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:55:29.875929 5570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:40287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:42829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:29.876703 5570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:29.877442 5570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:29.885491 5578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:29.885500 5575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:29.885577 5576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:29.885598 5570 server_base.cc:1034] running on GCE node
I20260619 14:55:29.886180 5570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:29.886762 5570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:29.887943 5570 hybrid_clock.cc:648] HybridClock initialized: now 1781880929887936 us; error 29 us; skew 500 ppm
Jun 19 14:55:29 dist-test-slave-3qdh krb5kdc[5195](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880929, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:29.890439 5570 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:55:29.891225 5570 webserver.cc:460] Webserver started at http://127.19.34.67:42777/ using document root <none> and password file <none>
I20260619 14:55:29.891636 5570 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:29.891685 5570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:29.891812 5570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:29.893546 5570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "56af5e0a75b2487f9975d3dc1fc94fe2"
format_stamp: "Formatted at 2026-06-19 14:55:29 on dist-test-slave-3qdh"
server_key: "9ab42e0777847bd22c64a119109d5a6e"
server_key_iv: "ba409ac1a389303fc2888bf2a373ff03"
server_key_version: "encryptionkey@0"
I20260619 14:55:29.894001 5570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "56af5e0a75b2487f9975d3dc1fc94fe2"
format_stamp: "Formatted at 2026-06-19 14:55:29 on dist-test-slave-3qdh"
server_key: "9ab42e0777847bd22c64a119109d5a6e"
server_key_iv: "ba409ac1a389303fc2888bf2a373ff03"
server_key_version: "encryptionkey@0"
I20260619 14:55:29.897086 5570 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:55:29.898986 5585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:29.899937 5570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:55:29.900009 5570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "56af5e0a75b2487f9975d3dc1fc94fe2"
format_stamp: "Formatted at 2026-06-19 14:55:29 on dist-test-slave-3qdh"
server_key: "9ab42e0777847bd22c64a119109d5a6e"
server_key_iv: "ba409ac1a389303fc2888bf2a373ff03"
server_key_version: "encryptionkey@0"
I20260619 14:55:29.900071 5570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:29.911044 5570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:29.913743 5570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:29.913918 5570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:29.914523 5570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:55:29.915236 5570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:55:29.915282 5570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:29.915316 5570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:55:29.915333 5570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:29.923331 5570 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:35313
I20260619 14:55:29.923378 5698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:35313 every 8 connection(s)
I20260619 14:55:29.924083 5570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:55:29.926615 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 5570
I20260619 14:55:29.926724 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:55:29.927044 19593 external_mini_cluster.cc:1428] Setting key b09e042d5dae51f8064e8b333ab77044
Jun 19 14:55:29 dist-test-slave-3qdh krb5kdc[5195](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880929, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:55:29.936213 5279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:29.925622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:60735 (local address 127.19.34.126:42829)
0619 14:55:29.925741 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:29.925744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:29.927354 (+ 1610us) server_negotiation.cc:407] Connection header received
0619 14:55:29.927946 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:29.927948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:29.927997 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:29.928098 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:29.929294 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.929781 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.930804 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.931004 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.931109 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:29.933259 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:29.933263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:29.933266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:29.933295 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:29.934420 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:29.934881 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:29.934885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:29.934887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:29.934960 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:29.935285 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:29.935288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:29.935290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:29.935445 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:29.935544 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:29.935952 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 14:55:29.936055 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:55:29.936874 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:29.925895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:42829 (local address 127.19.34.67:60735)
0619 14:55:29.926290 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:29.926326 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:29.927715 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:29.928245 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:29.928252 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:29.928683 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:29.929158 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:29.929169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.929902 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.929905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:29.930682 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:29.930690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.931136 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.931140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:29.931296 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:29.931739 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:29.931753 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:29.933099 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:29.934530 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:29.934535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:29.934546 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:29.934779 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:29.935046 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:29.935050 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:29.935052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:29.935177 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:29.935541 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:29.935546 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:29.935748 (+ 202us) client_negotiation.cc:769] Sending connection context
0619 14:55:29.935913 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 14:55:29.936161 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":93,"threads_started":1}
I20260619 14:55:29.937777 5699 heartbeater.cc:347] Connected to a master server at 127.19.34.126:42829
I20260619 14:55:29.937976 5699 heartbeater.cc:464] Registering TS with master...
I20260619 14:55:29.938340 5699 heartbeater.cc:511] Master 127.19.34.126:42829 requested a full tablet report, sending...
I20260619 14:55:29.939131 5240 ts_manager.cc:194] Registered new tserver with Master: 56af5e0a75b2487f9975d3dc1fc94fe2 (127.19.34.67:35313)
I20260619 14:55:29.939666 5240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:60735
I20260619 14:55:29.940718 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:55:29.948802 5279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:29.942604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57798 (local address 127.19.34.126:42829)
0619 14:55:29.942733 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:29.942737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:29.942907 (+ 170us) server_negotiation.cc:407] Connection header received
0619 14:55:29.942991 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:29.942993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:29.943025 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:29.943084 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:29.943694 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.944081 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.944844 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.945011 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.945075 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:29.946196 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:29.946199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:29.946200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:29.946219 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:29.947248 (+ 1029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:29.947637 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:29.947640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:29.947641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:29.947700 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:29.947989 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:29.947991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:29.947992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:29.948161 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:29.948265 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:29.948547 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 14:55:29.948643 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:55:29.951107 5240 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:55:29.953934 5240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:55:29.966013 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:29.961177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33660 (local address 127.19.34.66:39137)
0619 14:55:29.961468 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:29.961477 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:29.961497 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:55:29.961562 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:29.961565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:29.961727 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:29.961867 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:29.962334 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.963353 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.964568 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.965415 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.965511 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:29.965656 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:29.965791 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 14:55:29.965874 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":120,"threads_started":1}
I20260619 14:55:29.966007 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:29.961036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:39137 (local address 127.0.0.1:33660)
0619 14:55:29.961311 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:29.961344 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:29.961436 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:29.962069 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:29.962075 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:29.962097 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:29.962218 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:29.962224 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.963490 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.963496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:29.964444 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:29.964452 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.965519 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.965523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:29.965635 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:29.965669 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 14:55:29.965766 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 14:55:29.965871 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
I20260619 14:55:29.966813 5712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:29.961647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:35313 (local address 127.0.0.1:53816)
0619 14:55:29.961845 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:29.961858 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:29.961924 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:29.962707 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:29.962710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:29.962729 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:29.962821 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:29.962832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.964319 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.964322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:29.965457 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:29.965466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.966477 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.966481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:29.966560 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:29.966579 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:55:29.966649 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 14:55:29.966708 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
I20260619 14:55:29.966917 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:29.961755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53816 (local address 127.19.34.67:35313)
0619 14:55:29.962270 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:29.962280 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:29.962298 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:55:29.962348 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:29.962353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:29.962542 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:29.962979 (+ 437us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:29.963090 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.964157 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.965587 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.966354 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.966459 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:29.966565 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:29.966697 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 14:55:29.966775 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":105,"threads_started":1}
I20260619 14:55:29.967454 5714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:29.961949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:42289 (local address 127.0.0.1:54722)
0619 14:55:29.962818 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:29.962831 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:29.962919 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:29.963336 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:29.963339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:29.963356 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:29.963457 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:29.963465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.964725 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.964728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:29.965805 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:29.965814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.967128 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.967131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:29.967207 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:29.967226 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:55:29.967299 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 14:55:29.967356 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"thread_start_us":80,"threads_started":1}
I20260619 14:55:29.967847 5715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:29.962321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54722 (local address 127.19.34.65:42289)
0619 14:55:29.962729 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:29.962740 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:29.962930 (+ 190us) server_negotiation.cc:407] Connection header received
0619 14:55:29.963007 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:29.963013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:29.963198 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:29.963338 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:29.963542 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.964584 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.965927 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:29.966816 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:29.967111 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:29.967229 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:29.967470 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 14:55:29.967696 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":129,"threads_started":1}
I20260619 14:55:29.969043 5499 tablet_service.cc:1467] Processing CreateTablet for tablet 3b0c6d1b0ea24abcae456e6ca4513b41 (DEFAULT_TABLE table=test-table [id=e68b4087b5904bc28acb92186e45bbbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:55:29.969437 5633 tablet_service.cc:1467] Processing CreateTablet for tablet 3b0c6d1b0ea24abcae456e6ca4513b41 (DEFAULT_TABLE table=test-table [id=e68b4087b5904bc28acb92186e45bbbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:55:29.970005 5499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b0c6d1b0ea24abcae456e6ca4513b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:29.970384 5633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b0c6d1b0ea24abcae456e6ca4513b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:29.970438 5365 tablet_service.cc:1467] Processing CreateTablet for tablet 3b0c6d1b0ea24abcae456e6ca4513b41 (DEFAULT_TABLE table=test-table [id=e68b4087b5904bc28acb92186e45bbbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:55:29.971257 5365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b0c6d1b0ea24abcae456e6ca4513b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:29.975721 5716 tablet_bootstrap.cc:492] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9: Bootstrap starting.
I20260619 14:55:29.976914 5717 tablet_bootstrap.cc:492] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2: Bootstrap starting.
I20260619 14:55:29.977897 5718 tablet_bootstrap.cc:492] T 3b0c6d1b0ea24abcae456e6ca4513b41 P c72b05af8218482ab52f9f412208a040: Bootstrap starting.
I20260619 14:55:29.978654 5716 tablet_bootstrap.cc:654] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:29.978650 5717 tablet_bootstrap.cc:654] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:29.980005 5716 log.cc:824] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:29.980005 5717 log.cc:824] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:29.980108 5718 tablet_bootstrap.cc:654] T 3b0c6d1b0ea24abcae456e6ca4513b41 P c72b05af8218482ab52f9f412208a040: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:29.980863 5718 log.cc:824] T 3b0c6d1b0ea24abcae456e6ca4513b41 P c72b05af8218482ab52f9f412208a040: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:29.981814 5716 tablet_bootstrap.cc:492] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9: No bootstrap required, opened a new log
I20260619 14:55:29.981858 5717 tablet_bootstrap.cc:492] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2: No bootstrap required, opened a new log
I20260619 14:55:29.982010 5716 ts_tablet_manager.cc:1397] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260619 14:55:29.982010 5717 ts_tablet_manager.cc:1397] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260619 14:55:29.982722 5718 tablet_bootstrap.cc:492] T 3b0c6d1b0ea24abcae456e6ca4513b41 P c72b05af8218482ab52f9f412208a040: No bootstrap required, opened a new log
I20260619 14:55:29.982899 5718 ts_tablet_manager.cc:1397] T 3b0c6d1b0ea24abcae456e6ca4513b41 P c72b05af8218482ab52f9f412208a040: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:55:29.984941 5716 raft_consensus.cc:348] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e429d417c6b45568ca6910d9938c2e9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39137 } } peers { permanent_uuid: "56af5e0a75b2487f9975d3dc1fc94fe2" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35313 } } peers { permanent_uuid: "c72b05af8218482ab52f9f412208a040" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 42289 } }
I20260619 14:55:29.985180 5716 raft_consensus.cc:374] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:29.985229 5716 raft_consensus.cc:729] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e429d417c6b45568ca6910d9938c2e9, State: Initialized, Role: FOLLOWER
I20260619 14:55:29.985620 5716 consensus_queue.cc:260] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e429d417c6b45568ca6910d9938c2e9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39137 } } peers { permanent_uuid: "56af5e0a75b2487f9975d3dc1fc94fe2" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35313 } } peers { permanent_uuid: "c72b05af8218482ab52f9f412208a040" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 42289 } }
I20260619 14:55:29.985872 5717 raft_consensus.cc:348] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e429d417c6b45568ca6910d9938c2e9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39137 } } peers { permanent_uuid: "56af5e0a75b2487f9975d3dc1fc94fe2" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35313 } } peers { permanent_uuid: "c72b05af8218482ab52f9f412208a040" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 42289 } }
I20260619 14:55:29.986136 5717 raft_consensus.cc:374] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:29.986181 5565 heartbeater.cc:503] Master 127.19.34.126:42829 was elected leader, sending a full tablet report...
I20260619 14:55:29.986196 5717 raft_consensus.cc:729] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56af5e0a75b2487f9975d3dc1fc94fe2, State: Initialized, Role: FOLLOWER
I20260619 14:55:29.986447 5718 raft_consensus.cc:348] T 3b0c6d1b0ea24abcae456e6ca4513b41 P c72b05af8218482ab52f9f412208a040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e429d417c6b45568ca6910d9938c2e9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39137 } } peers { permanent_uuid: "56af5e0a75b2487f9975d3dc1fc94fe2" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35313 } } peers { permanent_uuid: "c72b05af8218482ab52f9f412208a040" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 42289 } }
I20260619 14:55:29.986604 5717 consensus_queue.cc:260] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e429d417c6b45568ca6910d9938c2e9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39137 } } peers { permanent_uuid: "56af5e0a75b2487f9975d3dc1fc94fe2" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35313 } } peers { permanent_uuid: "c72b05af8218482ab52f9f412208a040" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 42289 } }
I20260619 14:55:29.986707 5718 raft_consensus.cc:374] T 3b0c6d1b0ea24abcae456e6ca4513b41 P c72b05af8218482ab52f9f412208a040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:29.986761 5718 raft_consensus.cc:729] T 3b0c6d1b0ea24abcae456e6ca4513b41 P c72b05af8218482ab52f9f412208a040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c72b05af8218482ab52f9f412208a040, State: Initialized, Role: FOLLOWER
I20260619 14:55:29.987142 5699 heartbeater.cc:503] Master 127.19.34.126:42829 was elected leader, sending a full tablet report...
I20260619 14:55:29.987175 5718 consensus_queue.cc:260] T 3b0c6d1b0ea24abcae456e6ca4513b41 P c72b05af8218482ab52f9f412208a040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e429d417c6b45568ca6910d9938c2e9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39137 } } peers { permanent_uuid: "56af5e0a75b2487f9975d3dc1fc94fe2" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35313 } } peers { permanent_uuid: "c72b05af8218482ab52f9f412208a040" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 42289 } }
I20260619 14:55:29.987495 5717 ts_tablet_manager.cc:1428] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260619 14:55:29.988070 5718 ts_tablet_manager.cc:1428] T 3b0c6d1b0ea24abcae456e6ca4513b41 P c72b05af8218482ab52f9f412208a040: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:55:29.988406 5431 heartbeater.cc:503] Master 127.19.34.126:42829 was elected leader, sending a full tablet report...
I20260619 14:55:29.988808 5716 ts_tablet_manager.cc:1428] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
W20260619 14:55:30.048194 5432 tablet.cc:2369] T 3b0c6d1b0ea24abcae456e6ca4513b41 P c72b05af8218482ab52f9f412208a040: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:55:30.160645 5722 raft_consensus.cc:482] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:55:30.160808 5722 raft_consensus.cc:504] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e429d417c6b45568ca6910d9938c2e9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39137 } } peers { permanent_uuid: "56af5e0a75b2487f9975d3dc1fc94fe2" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35313 } } peers { permanent_uuid: "c72b05af8218482ab52f9f412208a040" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 42289 } }
I20260619 14:55:30.161691 5722 leader_election.cc:288] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56af5e0a75b2487f9975d3dc1fc94fe2 (127.19.34.67:35313), c72b05af8218482ab52f9f412208a040 (127.19.34.65:42289)
I20260619 14:55:30.165117 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:30.161898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:35313 (local address 127.19.34.66:60039)
0619 14:55:30.162034 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:30.162049 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:30.162133 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:30.162374 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:30.162377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:30.162392 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:30.162525 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:30.162531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.163261 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.163266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:30.164036 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:30.164044 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.164705 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.164708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:30.164821 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:30.164841 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:55:30.164944 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 14:55:30.165008 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 14:55:30.165117 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:30.162014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:60039 (local address 127.19.34.67:35313)
0619 14:55:30.162152 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:30.162156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:30.162172 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:55:30.162236 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:30.162241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:30.162287 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:30.162358 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:30.162643 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.163138 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.164162 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.164591 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.164718 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:30.164784 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:30.164958 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 14:55:30.165013 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:55:30.165514 5715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:30.162252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:34967 (local address 127.19.34.65:42289)
0619 14:55:30.162380 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:30.162389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:30.162548 (+ 159us) server_negotiation.cc:407] Connection header received
0619 14:55:30.162671 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:30.162674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:30.162733 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:30.162822 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:30.163098 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.163567 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.164587 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.164957 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.165078 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:30.165117 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:30.165344 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 14:55:30.165403 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 14:55:30.165719 5653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0c6d1b0ea24abcae456e6ca4513b41" candidate_uuid: "4e429d417c6b45568ca6910d9938c2e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56af5e0a75b2487f9975d3dc1fc94fe2" is_pre_election: true
I20260619 14:55:30.165839 5725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:30.162140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:42289 (local address 127.19.34.66:34967)
0619 14:55:30.162431 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:30.162460 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:30.162584 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:30.162833 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:30.162836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:30.162851 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:30.162983 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:30.162988 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.163710 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.163716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:30.164471 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:30.164479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.165071 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.165074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:30.165158 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:30.165192 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 14:55:30.165660 (+ 468us) client_negotiation.cc:240] Negotiation successful
0619 14:55:30.165730 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":94,"threads_started":1}
I20260619 14:55:30.166023 5653 raft_consensus.cc:2454] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e429d417c6b45568ca6910d9938c2e9 in term 0.
I20260619 14:55:30.166463 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0c6d1b0ea24abcae456e6ca4513b41" candidate_uuid: "4e429d417c6b45568ca6910d9938c2e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c72b05af8218482ab52f9f412208a040" is_pre_election: true
I20260619 14:55:30.166489 5455 leader_election.cc:302] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e429d417c6b45568ca6910d9938c2e9, 56af5e0a75b2487f9975d3dc1fc94fe2; no voters:
I20260619 14:55:30.166714 5385 raft_consensus.cc:2454] T 3b0c6d1b0ea24abcae456e6ca4513b41 P c72b05af8218482ab52f9f412208a040 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e429d417c6b45568ca6910d9938c2e9 in term 0.
I20260619 14:55:30.166749 5722 raft_consensus.cc:2781] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:55:30.166831 5722 raft_consensus.cc:482] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:55:30.166873 5722 raft_consensus.cc:3037] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:30.167914 5722 raft_consensus.cc:504] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e429d417c6b45568ca6910d9938c2e9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39137 } } peers { permanent_uuid: "56af5e0a75b2487f9975d3dc1fc94fe2" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35313 } } peers { permanent_uuid: "c72b05af8218482ab52f9f412208a040" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 42289 } }
I20260619 14:55:30.168242 5722 leader_election.cc:288] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 [CANDIDATE]: Term 1 election: Requested vote from peers 56af5e0a75b2487f9975d3dc1fc94fe2 (127.19.34.67:35313), c72b05af8218482ab52f9f412208a040 (127.19.34.65:42289)
I20260619 14:55:30.168641 5653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0c6d1b0ea24abcae456e6ca4513b41" candidate_uuid: "4e429d417c6b45568ca6910d9938c2e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56af5e0a75b2487f9975d3dc1fc94fe2"
I20260619 14:55:30.168648 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b0c6d1b0ea24abcae456e6ca4513b41" candidate_uuid: "4e429d417c6b45568ca6910d9938c2e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c72b05af8218482ab52f9f412208a040"
I20260619 14:55:30.168756 5653 raft_consensus.cc:3037] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:30.168756 5385 raft_consensus.cc:3037] T 3b0c6d1b0ea24abcae456e6ca4513b41 P c72b05af8218482ab52f9f412208a040 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:30.169883 5385 raft_consensus.cc:2454] T 3b0c6d1b0ea24abcae456e6ca4513b41 P c72b05af8218482ab52f9f412208a040 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e429d417c6b45568ca6910d9938c2e9 in term 1.
I20260619 14:55:30.169883 5653 raft_consensus.cc:2454] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e429d417c6b45568ca6910d9938c2e9 in term 1.
I20260619 14:55:30.170125 5455 leader_election.cc:302] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e429d417c6b45568ca6910d9938c2e9, 56af5e0a75b2487f9975d3dc1fc94fe2; no voters:
I20260619 14:55:30.170271 5722 raft_consensus.cc:2781] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:30.170481 5722 raft_consensus.cc:686] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 [term 1 LEADER]: Becoming Leader. State: Replica: 4e429d417c6b45568ca6910d9938c2e9, State: Running, Role: LEADER
I20260619 14:55:30.170786 5722 consensus_queue.cc:237] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e429d417c6b45568ca6910d9938c2e9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39137 } } peers { permanent_uuid: "56af5e0a75b2487f9975d3dc1fc94fe2" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35313 } } peers { permanent_uuid: "c72b05af8218482ab52f9f412208a040" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 42289 } }
I20260619 14:55:30.173506 5239 catalog_manager.cc:5495] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e429d417c6b45568ca6910d9938c2e9 (127.19.34.66). New cstate: current_term: 1 leader_uuid: "4e429d417c6b45568ca6910d9938c2e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e429d417c6b45568ca6910d9938c2e9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 39137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56af5e0a75b2487f9975d3dc1fc94fe2" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c72b05af8218482ab52f9f412208a040" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 42289 } health_report { overall_health: UNKNOWN } } }
W20260619 14:55:30.174860 5700 tablet.cc:2369] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:55:30.197564 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:30.194770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.19.34.66:39137)
0619 14:55:30.194875 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:30.194878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:30.194889 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:55:30.194986 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:30.194987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:30.195026 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:30.195095 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:30.195282 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.195661 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.196401 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.196571 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.196636 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:30.196878 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:30.197210 (+ 332us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:30.197308 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:30.197381 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 14:55:30.197432 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:55:30.202168 5385 raft_consensus.cc:1261] T 3b0c6d1b0ea24abcae456e6ca4513b41 P c72b05af8218482ab52f9f412208a040 [term 1 FOLLOWER]: Refusing update from remote peer 4e429d417c6b45568ca6910d9938c2e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:55:30.202169 5653 raft_consensus.cc:1261] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2 [term 1 FOLLOWER]: Refusing update from remote peer 4e429d417c6b45568ca6910d9938c2e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:55:30.202841 5726 consensus_queue.cc:1035] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c72b05af8218482ab52f9f412208a040" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 42289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:55:30.203027 5722 consensus_queue.cc:1035] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56af5e0a75b2487f9975d3dc1fc94fe2" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:55:30.208168 5729 mvcc.cc:205] Tried to move back new op lower bound from 7298584290102538240 to 7298584289981575168. Current Snapshot: MvccSnapshot[applied={T|T < 7298584290102538240}]
I20260619 14:55:30.209489 5730 mvcc.cc:205] Tried to move back new op lower bound from 7298584290102538240 to 7298584289981575168. Current Snapshot: MvccSnapshot[applied={T|T < 7298584290102538240}]
I20260619 14:55:30.217759 5239 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:55:30.217892 5239 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:55:30.219895 5239 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2423ece1751e4a3ab8c03f71bd667cd0: Sending DeleteTablet for 3 replicas of tablet 3b0c6d1b0ea24abcae456e6ca4513b41
I20260619 14:55:30.220690 5499 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b0c6d1b0ea24abcae456e6ca4513b41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:55:30 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:33660
I20260619 14:55:30.220695 5633 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b0c6d1b0ea24abcae456e6ca4513b41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:55:30 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:53816
I20260619 14:55:30.221097 5739 tablet_replica.cc:330] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2: stopping tablet replica
I20260619 14:55:30.221248 5365 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b0c6d1b0ea24abcae456e6ca4513b41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:55:30 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:54722
I20260619 14:55:30.221391 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 5302
I20260619 14:55:30.222882 5740 tablet_replica.cc:330] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9: stopping tablet replica
I20260619 14:55:30.223129 5740 raft_consensus.cc:2229] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:55:30.223382 5739 raft_consensus.cc:2229] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:55:30.223450 5740 raft_consensus.cc:2258] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:55:30.223742 5739 raft_consensus.cc:2258] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:55:30.224988 5739 ts_tablet_manager.cc:1905] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:55:30.225327 5740 ts_tablet_manager.cc:1905] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:55:30.227560 5739 ts_tablet_manager.cc:1918] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:55:30.227655 5739 log.cc:1196] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/3b0c6d1b0ea24abcae456e6ca4513b41
I20260619 14:55:30.227967 5740 ts_tablet_manager.cc:1918] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:55:30.228029 5739 ts_tablet_manager.cc:1939] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 56af5e0a75b2487f9975d3dc1fc94fe2: Deleting consensus metadata
I20260619 14:55:30.228142 5740 log.cc:1196] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880697016847-19593-0/minicluster-data/ts-1/wal/wals/3b0c6d1b0ea24abcae456e6ca4513b41
W20260619 14:55:30.228463 5227 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.34.65:42289 (error 108)
I20260619 14:55:30.228507 5740 ts_tablet_manager.cc:1939] T 3b0c6d1b0ea24abcae456e6ca4513b41 P 4e429d417c6b45568ca6910d9938c2e9: Deleting consensus metadata
I20260619 14:55:30.229175 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 5436
I20260619 14:55:30.229202 5227 catalog_manager.cc:4841] TS 56af5e0a75b2487f9975d3dc1fc94fe2 (127.19.34.67:35313): tablet 3b0c6d1b0ea24abcae456e6ca4513b41 (table test-table [id=e68b4087b5904bc28acb92186e45bbbe]) successfully deleted
I20260619 14:55:30.230581 5714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:30.230251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:42289 (local address 127.0.0.1:54730)
0619 14:55:30.230420 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:30.230482 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.65:42289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260619 14:55:30.230731 5227 catalog_manager.cc:4584] TS c72b05af8218482ab52f9f412208a040 (127.19.34.65:42289): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3b0c6d1b0ea24abcae456e6ca4513b41: Network error: Client connection negotiation failed: client connection to 127.19.34.65:42289: connect: Connection refused (error 111)
I20260619 14:55:30.234797 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 5570
I20260619 14:55:30.235445 5714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:30.234926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:39137 (local address 127.0.0.1:33680)
0619 14:55:30.235326 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:30.235377 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.66:39137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328}
I20260619 14:55:30.240120 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 5208
2026-06-19T14:55:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2159 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T14:55:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:30Z Disabled control of system clock
I20260619 14:55:30.267284 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:33105
--webserver_interface=127.19.34.126
--webserver_port=0
--builtin_ntp_servers=127.19.34.84:35621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:33105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:55:30.363044 5745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:30.363252 5745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:30.363286 5745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:30.366887 5745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:30.366927 5745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:30.366942 5745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:30.366957 5745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:30.366971 5745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:30.371575 5745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:33105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:33105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:30.372334 5745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:30.373430 5745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:30.380734 5751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:30.380752 5750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:30.380820 5745 server_base.cc:1034] running on GCE node
W20260619 14:55:30.380743 5753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:30.381469 5745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:30.382355 5745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:30.383597 5745 hybrid_clock.cc:648] HybridClock initialized: now 1781880930383584 us; error 32 us; skew 500 ppm
I20260619 14:55:30.385139 5745 webserver.cc:460] Webserver started at http://127.19.34.126:38523/ using document root <none> and password file <none>
I20260619 14:55:30.385608 5745 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:30.385703 5745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:30.385905 5745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:30.387609 5745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "1878338168c844e39c9701639efa928c"
format_stamp: "Formatted at 2026-06-19 14:55:30 on dist-test-slave-3qdh"
server_key: "38e8f3689ecc91b62e53ba1aa83bf002"
server_key_iv: "3fa087eb1aaf558205baf6c80e77948a"
server_key_version: "encryptionkey@0"
I20260619 14:55:30.388167 5745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "1878338168c844e39c9701639efa928c"
format_stamp: "Formatted at 2026-06-19 14:55:30 on dist-test-slave-3qdh"
server_key: "38e8f3689ecc91b62e53ba1aa83bf002"
server_key_iv: "3fa087eb1aaf558205baf6c80e77948a"
server_key_version: "encryptionkey@0"
I20260619 14:55:30.391360 5745 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 14:55:30.393474 5759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:30.394523 5745 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:55:30.394667 5745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "1878338168c844e39c9701639efa928c"
format_stamp: "Formatted at 2026-06-19 14:55:30 on dist-test-slave-3qdh"
server_key: "38e8f3689ecc91b62e53ba1aa83bf002"
server_key_iv: "3fa087eb1aaf558205baf6c80e77948a"
server_key_version: "encryptionkey@0"
I20260619 14:55:30.394769 5745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:30.409997 5745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:30.410483 5745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:30.410646 5745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:30.418952 5745 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:33105
I20260619 14:55:30.419035 5811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:33105 every 8 connection(s)
I20260619 14:55:30.420203 5745 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:30.421931 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 5745
I20260619 14:55:30.422041 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:30.422382 19593 external_mini_cluster.cc:1428] Setting key 12c2d942b4e6bb9c047990308211da28
I20260619 14:55:30.424226 5812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:30.431216 5812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c: Bootstrap starting.
I20260619 14:55:30.433061 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:30.423762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36168 (local address 127.19.34.126:33105)
0619 14:55:30.424694 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:30.424711 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:30.424766 (+ 55us) server_negotiation.cc:407] Connection header received
0619 14:55:30.425339 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:30.425354 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:30.425673 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:30.425954 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:30.426330 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.427828 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.429048 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.429872 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.430026 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:30.431044 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:30.431050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:30.431077 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:30.431521 (+ 444us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:30.431586 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:30.431610 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:30.431713 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:30.431863 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 14:55:30.432119 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":101,"threads_started":1}
I20260619 14:55:30.433825 5812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:30.434547 5812 log.cc:824] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:30.436774 5812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c: No bootstrap required, opened a new log
I20260619 14:55:30.441232 5812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1878338168c844e39c9701639efa928c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33105 } }
I20260619 14:55:30.441434 5812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:30.441486 5812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1878338168c844e39c9701639efa928c, State: Initialized, Role: FOLLOWER
I20260619 14:55:30.441869 5812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1878338168c844e39c9701639efa928c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33105 } }
I20260619 14:55:30.442013 5812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:30.442070 5812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:30.442147 5812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:30.442917 5812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1878338168c844e39c9701639efa928c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33105 } }
I20260619 14:55:30.443251 5812 leader_election.cc:302] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1878338168c844e39c9701639efa928c; no voters:
I20260619 14:55:30.443539 5812 leader_election.cc:288] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:30.443674 5817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:30.443907 5817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c [term 1 LEADER]: Becoming Leader. State: Replica: 1878338168c844e39c9701639efa928c, State: Running, Role: LEADER
I20260619 14:55:30.444144 5817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1878338168c844e39c9701639efa928c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33105 } }
I20260619 14:55:30.444748 5812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:30.445622 5818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1878338168c844e39c9701639efa928c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1878338168c844e39c9701639efa928c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33105 } } }
I20260619 14:55:30.445744 5818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:30.446062 5826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:30.446054 5819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1878338168c844e39c9701639efa928c. Latest consensus state: current_term: 1 leader_uuid: "1878338168c844e39c9701639efa928c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1878338168c844e39c9701639efa928c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33105 } } }
I20260619 14:55:30.446667 5819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:30.449144 5826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:30.454303 5826 catalog_manager.cc:1347] Generated new cluster ID: 93ff38b7b0c44a6bbf462e2af735db60
I20260619 14:55:30.454360 5826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:30.459918 5826 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:30.460690 5826 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:30.465855 5826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c: Generated new TSK 0
I20260619 14:55:30.466367 5826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:30.472990 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:33105
--builtin_ntp_servers=127.19.34.84:35621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:55:30.567365 5836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:30.567571 5836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:30.567603 5836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:30.571309 5836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:30.571352 5836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:30.571399 5836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:55:30.575691 5836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:33105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:30.576525 5836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:30.577826 5836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:30.586118 5841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:30.586226 5836 server_base.cc:1034] running on GCE node
W20260619 14:55:30.586155 5842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:30.586118 5844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:30.586743 5836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:30.587275 5836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:30.588454 5836 hybrid_clock.cc:648] HybridClock initialized: now 1781880930588439 us; error 25 us; skew 500 ppm
I20260619 14:55:30.590045 5836 webserver.cc:460] Webserver started at http://127.19.34.65:38927/ using document root <none> and password file <none>
I20260619 14:55:30.590514 5836 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:30.590598 5836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:30.590776 5836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:30.592566 5836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "e0a2b15a443e45b0961b08aca9267fe7"
format_stamp: "Formatted at 2026-06-19 14:55:30 on dist-test-slave-3qdh"
server_key: "d35cdd9b6393e37148ca8bc71cc4f285"
server_key_iv: "50a525cfafd36b9948f80406ebfd7e87"
server_key_version: "encryptionkey@0"
I20260619 14:55:30.593086 5836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "e0a2b15a443e45b0961b08aca9267fe7"
format_stamp: "Formatted at 2026-06-19 14:55:30 on dist-test-slave-3qdh"
server_key: "d35cdd9b6393e37148ca8bc71cc4f285"
server_key_iv: "50a525cfafd36b9948f80406ebfd7e87"
server_key_version: "encryptionkey@0"
I20260619 14:55:30.596086 5836 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:55:30.598274 5850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:30.599323 5836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:30.599447 5836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "e0a2b15a443e45b0961b08aca9267fe7"
format_stamp: "Formatted at 2026-06-19 14:55:30 on dist-test-slave-3qdh"
server_key: "d35cdd9b6393e37148ca8bc71cc4f285"
server_key_iv: "50a525cfafd36b9948f80406ebfd7e87"
server_key_version: "encryptionkey@0"
I20260619 14:55:30.599545 5836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:30.621194 5836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:30.621665 5836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:30.621883 5836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:30.622346 5836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:55:30.623168 5836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:55:30.623245 5836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:30.623308 5836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:55:30.623364 5836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:30.630980 5836 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:34487
I20260619 14:55:30.631004 5963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:34487 every 8 connection(s)
I20260619 14:55:30.631633 5836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:55:30.638237 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 5836
I20260619 14:55:30.638374 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:55:30.638690 19593 external_mini_cluster.cc:1428] Setting key f976f7b149b9c95b62e0a1ed36eed8af
I20260619 14:55:30.639467 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:30.633202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:41773 (local address 127.19.34.126:33105)
0619 14:55:30.633326 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:30.633330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:30.634149 (+ 819us) server_negotiation.cc:407] Connection header received
0619 14:55:30.635291 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:30.635294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:30.635348 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:30.635440 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:30.636153 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.636634 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.637576 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.637772 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.637883 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:30.638886 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:30.638889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:30.638891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:30.638901 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:30.638918 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:30.638925 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:30.638980 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:30.639286 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 14:55:30.639355 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:55:30.640177 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:30.633453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:33105 (local address 127.19.34.65:41773)
0619 14:55:30.634010 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:30.634040 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:30.635112 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:30.635580 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:30.635585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:30.635658 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:30.636016 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:30.636026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.636752 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.636755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:30.637458 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:30.637466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.637934 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.637938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:30.638103 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:30.638628 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:55:30.638630 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:30.638674 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:55:30.638741 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:30.639027 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:30.639036 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:30.639043 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:55:30.639174 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 14:55:30.639376 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":121,"threads_started":1}
I20260619 14:55:30.640746 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:33105
--builtin_ntp_servers=127.19.34.84:35621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:55:30.641113 5964 heartbeater.cc:347] Connected to a master server at 127.19.34.126:33105
I20260619 14:55:30.641301 5964 heartbeater.cc:464] Registering TS with master...
I20260619 14:55:30.641692 5964 heartbeater.cc:511] Master 127.19.34.126:33105 requested a full tablet report, sending...
I20260619 14:55:30.643028 5776 ts_manager.cc:194] Registered new tserver with Master: e0a2b15a443e45b0961b08aca9267fe7 (127.19.34.65:34487)
I20260619 14:55:30.644069 5776 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.34.65:41773
W20260619 14:55:30.736397 5967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:30.736634 5967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:30.736688 5967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:30.739985 5967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:30.740060 5967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:30.740139 5967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:55:30.744234 5967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:33105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:30.745034 5967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:30.746178 5967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:30.754302 5975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:30.754331 5972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:30.754377 5967 server_base.cc:1034] running on GCE node
W20260619 14:55:30.754304 5973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:30.754861 5967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:30.755311 5967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:30.756505 5967 hybrid_clock.cc:648] HybridClock initialized: now 1781880930756489 us; error 25 us; skew 500 ppm
I20260619 14:55:30.757894 5967 webserver.cc:460] Webserver started at http://127.19.34.66:42453/ using document root <none> and password file <none>
I20260619 14:55:30.758394 5967 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:30.758479 5967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:30.758674 5967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:30.760412 5967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "c6ecccf94d44433b9704a9d82cb993c1"
format_stamp: "Formatted at 2026-06-19 14:55:30 on dist-test-slave-3qdh"
server_key: "103977abb9f5869aadb74b3c3d722b86"
server_key_iv: "90db4bda89284a2cd9e990b9c90655a7"
server_key_version: "encryptionkey@0"
I20260619 14:55:30.760895 5967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "c6ecccf94d44433b9704a9d82cb993c1"
format_stamp: "Formatted at 2026-06-19 14:55:30 on dist-test-slave-3qdh"
server_key: "103977abb9f5869aadb74b3c3d722b86"
server_key_iv: "90db4bda89284a2cd9e990b9c90655a7"
server_key_version: "encryptionkey@0"
I20260619 14:55:30.764005 5967 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:55:30.766117 5981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:30.767066 5967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:30.767197 5967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "c6ecccf94d44433b9704a9d82cb993c1"
format_stamp: "Formatted at 2026-06-19 14:55:30 on dist-test-slave-3qdh"
server_key: "103977abb9f5869aadb74b3c3d722b86"
server_key_iv: "90db4bda89284a2cd9e990b9c90655a7"
server_key_version: "encryptionkey@0"
I20260619 14:55:30.767297 5967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:30.782984 5967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:30.783521 5967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:30.783703 5967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:30.784194 5967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:55:30.784967 5967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:55:30.785039 5967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:30.785097 5967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:55:30.785130 5967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:30.792888 5967 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:46491
I20260619 14:55:30.792917 6094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:46491 every 8 connection(s)
I20260619 14:55:30.793645 5967 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:55:30.796629 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 5967
I20260619 14:55:30.796743 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:55:30.797061 19593 external_mini_cluster.cc:1428] Setting key 3a135d8193dfacb0879d6116175801ac
I20260619 14:55:30.798879 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:33105
--builtin_ntp_servers=127.19.34.84:35621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:55:30.802901 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:30.795224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:45721 (local address 127.19.34.126:33105)
0619 14:55:30.795358 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:30.795363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:30.796269 (+ 906us) server_negotiation.cc:407] Connection header received
0619 14:55:30.798108 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:30.798112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:30.798159 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:30.798230 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:30.799205 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.799659 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.800887 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.801177 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.801251 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:30.802373 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:30.802379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:30.802384 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:30.802399 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:30.802422 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:30.802430 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:30.802517 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:30.802715 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 14:55:30.802758 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:55:30.804055 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:30.795576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:33105 (local address 127.19.34.66:45721)
0619 14:55:30.796137 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:30.796177 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:30.797539 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:30.798399 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:30.798407 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:30.798496 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:30.799065 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:30.799084 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.799778 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.799784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:30.800776 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:30.800784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.801313 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.801319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:30.801508 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:30.802075 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:55:30.802079 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:30.802127 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:55:30.802208 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:30.802554 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:30.802565 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:30.802575 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 14:55:30.802710 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 14:55:30.803060 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":96,"threads_started":1}
I20260619 14:55:30.805212 6095 heartbeater.cc:347] Connected to a master server at 127.19.34.126:33105
I20260619 14:55:30.805442 6095 heartbeater.cc:464] Registering TS with master...
I20260619 14:55:30.805892 6095 heartbeater.cc:511] Master 127.19.34.126:33105 requested a full tablet report, sending...
I20260619 14:55:30.806881 5776 ts_manager.cc:194] Registered new tserver with Master: c6ecccf94d44433b9704a9d82cb993c1 (127.19.34.66:46491)
I20260619 14:55:30.807420 5776 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.34.66:45721
W20260619 14:55:30.900380 6098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:30.900620 6098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:30.900684 6098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:30.904444 6098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:30.904506 6098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:30.904587 6098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:55:30.908739 6098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:33105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:30.909538 6098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:30.910707 6098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:30.918956 6106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:30.918978 6103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:30.918975 6104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:30.919210 6098 server_base.cc:1034] running on GCE node
I20260619 14:55:30.919524 6098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:30.920061 6098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:30.921233 6098 hybrid_clock.cc:648] HybridClock initialized: now 1781880930921226 us; error 27 us; skew 500 ppm
I20260619 14:55:30.922667 6098 webserver.cc:460] Webserver started at http://127.19.34.67:34085/ using document root <none> and password file <none>
I20260619 14:55:30.923170 6098 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:30.923261 6098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:30.923470 6098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:30.925202 6098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "14c6d4f4f4e04d4c8acb0f4c97246aa6"
format_stamp: "Formatted at 2026-06-19 14:55:30 on dist-test-slave-3qdh"
server_key: "69db88097cf268fee90f3ac6b0cc76d0"
server_key_iv: "19208276d0d1d242b6bec2e4edd61ac7"
server_key_version: "encryptionkey@0"
I20260619 14:55:30.925706 6098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "14c6d4f4f4e04d4c8acb0f4c97246aa6"
format_stamp: "Formatted at 2026-06-19 14:55:30 on dist-test-slave-3qdh"
server_key: "69db88097cf268fee90f3ac6b0cc76d0"
server_key_iv: "19208276d0d1d242b6bec2e4edd61ac7"
server_key_version: "encryptionkey@0"
I20260619 14:55:30.928787 6098 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:55:30.930747 6112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:30.931762 6098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:30.931882 6098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "14c6d4f4f4e04d4c8acb0f4c97246aa6"
format_stamp: "Formatted at 2026-06-19 14:55:30 on dist-test-slave-3qdh"
server_key: "69db88097cf268fee90f3ac6b0cc76d0"
server_key_iv: "19208276d0d1d242b6bec2e4edd61ac7"
server_key_version: "encryptionkey@0"
I20260619 14:55:30.931980 6098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:30.940579 6098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:30.941076 6098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:30.941249 6098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:30.941712 6098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:55:30.942442 6098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:55:30.942554 6098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:30.942615 6098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:55:30.942679 6098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:30.950845 6098 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:39065
I20260619 14:55:30.950861 6225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:39065 every 8 connection(s)
I20260619 14:55:30.951521 6098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:55:30.954306 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 6098
I20260619 14:55:30.954401 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:55:30.954651 19593 external_mini_cluster.cc:1428] Setting key 43f1a22356d842d4c32510ec9ae65cfa
I20260619 14:55:30.959599 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:30.953084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:58883 (local address 127.19.34.126:33105)
0619 14:55:30.953237 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:30.953242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:30.954097 (+ 855us) server_negotiation.cc:407] Connection header received
0619 14:55:30.955630 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:30.955633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:30.955706 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:30.955779 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:30.956694 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.957181 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.958076 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.958253 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.958318 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:30.959269 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:30.959271 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:30.959273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:30.959282 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:30.959294 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:30.959300 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:30.959347 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:30.959467 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 14:55:30.959507 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:55:30.960275 6228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:30.953414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:33105 (local address 127.19.34.67:58883)
0619 14:55:30.953977 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:30.954007 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:30.955458 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:30.955950 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:30.955957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:30.956039 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:30.956553 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:30.956569 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.957275 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.957277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:30.957983 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:30.957991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.958412 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.958415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:30.958536 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:30.959069 (+ 533us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:55:30.959071 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:30.959106 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:55:30.959156 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:30.959365 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:30.959370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:30.959374 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:55:30.959451 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 14:55:30.959671 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":95,"threads_started":1}
I20260619 14:55:30.961275 6226 heartbeater.cc:347] Connected to a master server at 127.19.34.126:33105
I20260619 14:55:30.961472 6226 heartbeater.cc:464] Registering TS with master...
I20260619 14:55:30.961890 6226 heartbeater.cc:511] Master 127.19.34.126:33105 requested a full tablet report, sending...
I20260619 14:55:30.962780 5776 ts_manager.cc:194] Registered new tserver with Master: 14c6d4f4f4e04d4c8acb0f4c97246aa6 (127.19.34.67:39065)
I20260619 14:55:30.963346 5776 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.34.67:58883
I20260619 14:55:30.967950 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:55:30.973285 5815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:30.969820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36170 (local address 127.19.34.126:33105)
0619 14:55:30.969952 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:30.969955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:30.970098 (+ 143us) server_negotiation.cc:407] Connection header received
0619 14:55:30.970220 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:30.970222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:30.970256 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:30.970348 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:30.970569 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.971091 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.971841 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.972023 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.972098 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:30.972910 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:30.972913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:30.972915 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:30.972925 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:30.972940 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:30.972948 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:30.973036 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:30.973167 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 14:55:30.973193 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:55:30.975893 5776 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:55:30.978726 5776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:55:30.991391 6240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:30.986059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:39065 (local address 127.0.0.1:49058)
0619 14:55:30.986700 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:30.986714 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:30.986788 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:30.987320 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:30.987325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:30.987371 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:30.987495 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:30.987501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.988731 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.988737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:30.990017 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:30.990026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.990918 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.990923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:30.991023 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:30.991053 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 14:55:30.991159 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 14:55:30.991255 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":69,"thread_start_us":96,"threads_started":1}
I20260619 14:55:30.991415 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:30.986385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49058 (local address 127.19.34.67:39065)
0619 14:55:30.986873 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:30.986888 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:30.986907 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:55:30.986956 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:30.986984 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:30.987168 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:30.987332 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:30.987612 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.988626 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.990141 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.990802 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.990895 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:30.991036 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:30.991188 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 14:55:30.991265 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":121,"threads_started":1}
I20260619 14:55:30.991698 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:30.986347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36250 (local address 127.19.34.65:34487)
0619 14:55:30.987028 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:30.987038 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:30.987063 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:55:30.987114 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:30.987116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:30.987619 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:30.987761 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:30.987986 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.989034 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.990261 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.990893 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.990977 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:30.991133 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:30.991451 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 14:55:30.991533 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":107,"threads_started":1}
I20260619 14:55:30.991763 6238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:30.985783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34487 (local address 127.0.0.1:36250)
0619 14:55:30.986373 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:30.986385 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:30.986506 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:30.987741 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:30.987743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:30.987758 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:30.987874 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:30.987883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.989179 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.989182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:30.990161 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:30.990168 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.991183 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.991186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:30.991251 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:30.991267 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 14:55:30.991324 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 14:55:30.991369 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":2,"spinlock_wait_cycles":12544,"thread_start_us":72,"threads_started":1}
I20260619 14:55:30.992143 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:30.985701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:46491 (local address 127.0.0.1:58150)
0619 14:55:30.986344 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:30.986385 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:30.986504 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:30.988174 (+ 1670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:30.988177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:30.988194 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:30.988272 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:30.988277 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.989537 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.989540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:30.990552 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:30.990583 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.991667 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.991672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:30.991772 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:30.991800 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 14:55:30.991964 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 14:55:30.992024 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":2,"spinlock_wait_cycles":2816,"thread_start_us":60,"threads_started":1}
I20260619 14:55:30.992252 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:30.985801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58150 (local address 127.19.34.66:46491)
0619 14:55:30.987600 (+ 1799us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:30.987610 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:30.987629 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:55:30.987693 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:30.987697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:30.987884 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:30.988036 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:30.988407 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.989386 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.990672 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:30.991337 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:30.991428 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:30.991579 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:30.992013 (+ 434us) server_negotiation.cc:299] Negotiation successful
0619 14:55:30.992096 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1689,"thread_start_us":89,"threads_started":1}
I20260619 14:55:30.994203 6160 tablet_service.cc:1467] Processing CreateTablet for tablet de91942477d04c10bf563460db17057a (DEFAULT_TABLE table=test-table [id=20e305c474714135b5f23f0fe2821452]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:55:30.994388 5898 tablet_service.cc:1467] Processing CreateTablet for tablet de91942477d04c10bf563460db17057a (DEFAULT_TABLE table=test-table [id=20e305c474714135b5f23f0fe2821452]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:55:30.994387 6029 tablet_service.cc:1467] Processing CreateTablet for tablet de91942477d04c10bf563460db17057a (DEFAULT_TABLE table=test-table [id=20e305c474714135b5f23f0fe2821452]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:55:30.995054 6160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de91942477d04c10bf563460db17057a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:30.995219 5898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de91942477d04c10bf563460db17057a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:30.995220 6029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de91942477d04c10bf563460db17057a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:31.001617 6244 tablet_bootstrap.cc:492] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1: Bootstrap starting.
I20260619 14:55:31.003294 6244 tablet_bootstrap.cc:654] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:31.003973 6244 log.cc:824] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:31.004527 6243 tablet_bootstrap.cc:492] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6: Bootstrap starting.
I20260619 14:55:31.004551 6245 tablet_bootstrap.cc:492] T de91942477d04c10bf563460db17057a P e0a2b15a443e45b0961b08aca9267fe7: Bootstrap starting.
I20260619 14:55:31.005681 6244 tablet_bootstrap.cc:492] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1: No bootstrap required, opened a new log
I20260619 14:55:31.005859 6244 ts_tablet_manager.cc:1397] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 14:55:31.006562 6245 tablet_bootstrap.cc:654] T de91942477d04c10bf563460db17057a P e0a2b15a443e45b0961b08aca9267fe7: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:31.006573 6243 tablet_bootstrap.cc:654] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:31.007411 6245 log.cc:824] T de91942477d04c10bf563460db17057a P e0a2b15a443e45b0961b08aca9267fe7: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:31.007409 6243 log.cc:824] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:31.009135 6243 tablet_bootstrap.cc:492] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6: No bootstrap required, opened a new log
I20260619 14:55:31.009330 6243 ts_tablet_manager.cc:1397] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:55:31.009327 6245 tablet_bootstrap.cc:492] T de91942477d04c10bf563460db17057a P e0a2b15a443e45b0961b08aca9267fe7: No bootstrap required, opened a new log
I20260619 14:55:31.009502 6245 ts_tablet_manager.cc:1397] T de91942477d04c10bf563460db17057a P e0a2b15a443e45b0961b08aca9267fe7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:55:31.009475 6244 raft_consensus.cc:348] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ecccf94d44433b9704a9d82cb993c1" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 46491 } } peers { permanent_uuid: "e0a2b15a443e45b0961b08aca9267fe7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34487 } } peers { permanent_uuid: "14c6d4f4f4e04d4c8acb0f4c97246aa6" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 39065 } }
I20260619 14:55:31.009716 6244 raft_consensus.cc:374] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:31.009780 6244 raft_consensus.cc:729] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ecccf94d44433b9704a9d82cb993c1, State: Initialized, Role: FOLLOWER
I20260619 14:55:31.010192 6244 consensus_queue.cc:260] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ecccf94d44433b9704a9d82cb993c1" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 46491 } } peers { permanent_uuid: "e0a2b15a443e45b0961b08aca9267fe7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34487 } } peers { permanent_uuid: "14c6d4f4f4e04d4c8acb0f4c97246aa6" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 39065 } }
I20260619 14:55:31.011020 6244 ts_tablet_manager.cc:1428] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:55:31.011354 6095 heartbeater.cc:503] Master 127.19.34.126:33105 was elected leader, sending a full tablet report...
I20260619 14:55:31.012887 6245 raft_consensus.cc:348] T de91942477d04c10bf563460db17057a P e0a2b15a443e45b0961b08aca9267fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ecccf94d44433b9704a9d82cb993c1" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 46491 } } peers { permanent_uuid: "e0a2b15a443e45b0961b08aca9267fe7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34487 } } peers { permanent_uuid: "14c6d4f4f4e04d4c8acb0f4c97246aa6" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 39065 } }
I20260619 14:55:31.013119 6245 raft_consensus.cc:374] T de91942477d04c10bf563460db17057a P e0a2b15a443e45b0961b08aca9267fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:31.013167 6245 raft_consensus.cc:729] T de91942477d04c10bf563460db17057a P e0a2b15a443e45b0961b08aca9267fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a2b15a443e45b0961b08aca9267fe7, State: Initialized, Role: FOLLOWER
I20260619 14:55:31.013181 6243 raft_consensus.cc:348] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ecccf94d44433b9704a9d82cb993c1" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 46491 } } peers { permanent_uuid: "e0a2b15a443e45b0961b08aca9267fe7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34487 } } peers { permanent_uuid: "14c6d4f4f4e04d4c8acb0f4c97246aa6" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 39065 } }
I20260619 14:55:31.013453 6243 raft_consensus.cc:374] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:31.013517 6243 raft_consensus.cc:729] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c6d4f4f4e04d4c8acb0f4c97246aa6, State: Initialized, Role: FOLLOWER
I20260619 14:55:31.013584 6245 consensus_queue.cc:260] T de91942477d04c10bf563460db17057a P e0a2b15a443e45b0961b08aca9267fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ecccf94d44433b9704a9d82cb993c1" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 46491 } } peers { permanent_uuid: "e0a2b15a443e45b0961b08aca9267fe7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34487 } } peers { permanent_uuid: "14c6d4f4f4e04d4c8acb0f4c97246aa6" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 39065 } }
I20260619 14:55:31.013973 6243 consensus_queue.cc:260] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ecccf94d44433b9704a9d82cb993c1" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 46491 } } peers { permanent_uuid: "e0a2b15a443e45b0961b08aca9267fe7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34487 } } peers { permanent_uuid: "14c6d4f4f4e04d4c8acb0f4c97246aa6" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 39065 } }
I20260619 14:55:31.014335 5964 heartbeater.cc:503] Master 127.19.34.126:33105 was elected leader, sending a full tablet report...
I20260619 14:55:31.014488 6245 ts_tablet_manager.cc:1428] T de91942477d04c10bf563460db17057a P e0a2b15a443e45b0961b08aca9267fe7: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260619 14:55:31.014772 6243 ts_tablet_manager.cc:1428] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260619 14:55:31.015051 6226 heartbeater.cc:503] Master 127.19.34.126:33105 was elected leader, sending a full tablet report...
W20260619 14:55:31.044461 6096 tablet.cc:2369] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:55:31.132659 5965 tablet.cc:2369] T de91942477d04c10bf563460db17057a P e0a2b15a443e45b0961b08aca9267fe7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:55:31.194598 6251 raft_consensus.cc:482] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:55:31.194813 6251 raft_consensus.cc:504] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ecccf94d44433b9704a9d82cb993c1" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 46491 } } peers { permanent_uuid: "e0a2b15a443e45b0961b08aca9267fe7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34487 } } peers { permanent_uuid: "14c6d4f4f4e04d4c8acb0f4c97246aa6" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 39065 } }
I20260619 14:55:31.195785 6251 leader_election.cc:288] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ecccf94d44433b9704a9d82cb993c1 (127.19.34.66:46491), e0a2b15a443e45b0961b08aca9267fe7 (127.19.34.65:34487)
I20260619 14:55:31.199420 6228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:31.195980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:46491 (local address 127.19.34.67:56439)
0619 14:55:31.196113 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:31.196128 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:31.196213 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:31.196639 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:31.196644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:31.196662 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:31.196770 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:31.196775 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:31.197524 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:31.197527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:31.198393 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:31.198401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:31.199010 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:31.199014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:31.199109 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:31.199134 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:55:31.199220 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 14:55:31.199292 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 14:55:31.199420 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:31.196245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:56439 (local address 127.19.34.66:46491)
0619 14:55:31.196392 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:31.196396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:31.196409 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:55:31.196456 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:31.196458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:31.196520 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:31.196607 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:31.196901 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:31.197420 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:31.198509 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:31.198891 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:31.199018 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:31.199064 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:31.199246 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 14:55:31.199320 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:55:31.199651 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:31.196129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:46333 (local address 127.19.34.65:34487)
0619 14:55:31.196267 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:31.196271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:31.196590 (+ 319us) server_negotiation.cc:407] Connection header received
0619 14:55:31.196721 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:31.196730 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:31.196768 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:31.196821 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:31.197098 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:31.197627 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:31.198714 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:31.199095 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:31.199242 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:31.199286 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:31.199462 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 14:55:31.199544 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:55:31.199913 6252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:31.195980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34487 (local address 127.19.34.67:46333)
0619 14:55:31.196504 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:31.196514 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:31.196622 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:31.196856 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:31.196862 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:31.196882 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:31.196999 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:31.197002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:31.197754 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:31.197760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:31.198580 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:31.198589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:31.199236 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:31.199239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:31.199339 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:31.199359 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:55:31.199449 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 14:55:31.199786 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":214,"thread_start_us":130,"threads_started":1}
I20260619 14:55:31.200049 6049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de91942477d04c10bf563460db17057a" candidate_uuid: "14c6d4f4f4e04d4c8acb0f4c97246aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ecccf94d44433b9704a9d82cb993c1" is_pre_election: true
I20260619 14:55:31.200335 6049 raft_consensus.cc:2454] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14c6d4f4f4e04d4c8acb0f4c97246aa6 in term 0.
I20260619 14:55:31.200465 5918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de91942477d04c10bf563460db17057a" candidate_uuid: "14c6d4f4f4e04d4c8acb0f4c97246aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0a2b15a443e45b0961b08aca9267fe7" is_pre_election: true
I20260619 14:55:31.200712 5918 raft_consensus.cc:2454] T de91942477d04c10bf563460db17057a P e0a2b15a443e45b0961b08aca9267fe7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14c6d4f4f4e04d4c8acb0f4c97246aa6 in term 0.
I20260619 14:55:31.200804 6115 leader_election.cc:302] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14c6d4f4f4e04d4c8acb0f4c97246aa6, c6ecccf94d44433b9704a9d82cb993c1; no voters:
I20260619 14:55:31.201026 6251 raft_consensus.cc:2781] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:55:31.201100 6251 raft_consensus.cc:482] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:55:31.201162 6251 raft_consensus.cc:3037] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:31.202109 6251 raft_consensus.cc:504] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ecccf94d44433b9704a9d82cb993c1" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 46491 } } peers { permanent_uuid: "e0a2b15a443e45b0961b08aca9267fe7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34487 } } peers { permanent_uuid: "14c6d4f4f4e04d4c8acb0f4c97246aa6" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 39065 } }
W20260619 14:55:31.202316 6227 tablet.cc:2369] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:55:31.202481 6251 leader_election.cc:288] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 [CANDIDATE]: Term 1 election: Requested vote from peers c6ecccf94d44433b9704a9d82cb993c1 (127.19.34.66:46491), e0a2b15a443e45b0961b08aca9267fe7 (127.19.34.65:34487)
I20260619 14:55:31.202797 5918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de91942477d04c10bf563460db17057a" candidate_uuid: "14c6d4f4f4e04d4c8acb0f4c97246aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0a2b15a443e45b0961b08aca9267fe7"
I20260619 14:55:31.202843 6049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de91942477d04c10bf563460db17057a" candidate_uuid: "14c6d4f4f4e04d4c8acb0f4c97246aa6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ecccf94d44433b9704a9d82cb993c1"
I20260619 14:55:31.202913 5918 raft_consensus.cc:3037] T de91942477d04c10bf563460db17057a P e0a2b15a443e45b0961b08aca9267fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:31.202996 6049 raft_consensus.cc:3037] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:31.203922 6049 raft_consensus.cc:2454] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c6d4f4f4e04d4c8acb0f4c97246aa6 in term 1.
I20260619 14:55:31.203989 5918 raft_consensus.cc:2454] T de91942477d04c10bf563460db17057a P e0a2b15a443e45b0961b08aca9267fe7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14c6d4f4f4e04d4c8acb0f4c97246aa6 in term 1.
I20260619 14:55:31.204200 6115 leader_election.cc:302] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14c6d4f4f4e04d4c8acb0f4c97246aa6, c6ecccf94d44433b9704a9d82cb993c1; no voters:
I20260619 14:55:31.204377 6251 raft_consensus.cc:2781] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:31.204546 6251 raft_consensus.cc:686] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 [term 1 LEADER]: Becoming Leader. State: Replica: 14c6d4f4f4e04d4c8acb0f4c97246aa6, State: Running, Role: LEADER
I20260619 14:55:31.204849 6251 consensus_queue.cc:237] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ecccf94d44433b9704a9d82cb993c1" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 46491 } } peers { permanent_uuid: "e0a2b15a443e45b0961b08aca9267fe7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34487 } } peers { permanent_uuid: "14c6d4f4f4e04d4c8acb0f4c97246aa6" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 39065 } }
I20260619 14:55:31.207583 5775 catalog_manager.cc:5495] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14c6d4f4f4e04d4c8acb0f4c97246aa6 (127.19.34.67). New cstate: current_term: 1 leader_uuid: "14c6d4f4f4e04d4c8acb0f4c97246aa6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ecccf94d44433b9704a9d82cb993c1" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 46491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0a2b15a443e45b0961b08aca9267fe7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14c6d4f4f4e04d4c8acb0f4c97246aa6" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 39065 } health_report { overall_health: HEALTHY } } }
I20260619 14:55:31.225323 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:31.222083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49066 (local address 127.19.34.67:39065)
0619 14:55:31.222180 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:31.222184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:31.222216 (+ 32us) server_negotiation.cc:407] Connection header received
0619 14:55:31.222343 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:31.222345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:31.222391 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:31.222468 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:31.222724 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:31.223190 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:31.224060 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:31.224287 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:31.224410 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:31.224680 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:31.224994 (+ 314us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:31.225115 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:31.225208 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 14:55:31.225232 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:55:31.230199 6049 raft_consensus.cc:1261] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1 [term 1 FOLLOWER]: Refusing update from remote peer 14c6d4f4f4e04d4c8acb0f4c97246aa6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:55:31.230239 5918 raft_consensus.cc:1261] T de91942477d04c10bf563460db17057a P e0a2b15a443e45b0961b08aca9267fe7 [term 1 FOLLOWER]: Refusing update from remote peer 14c6d4f4f4e04d4c8acb0f4c97246aa6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:55:31.230754 6253 consensus_queue.cc:1035] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ecccf94d44433b9704a9d82cb993c1" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 46491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:55:31.231030 6251 consensus_queue.cc:1035] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0a2b15a443e45b0961b08aca9267fe7" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:55:31.244673 5776 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:55:31.244858 5776 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:55:31.247097 5776 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1878338168c844e39c9701639efa928c: Sending DeleteTablet for 3 replicas of tablet de91942477d04c10bf563460db17057a
I20260619 14:55:31.247764 6029 tablet_service.cc:1514] Processing DeleteTablet for tablet de91942477d04c10bf563460db17057a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:55:31 UTC) from {username='slave'} at 127.0.0.1:58150
I20260619 14:55:31.247963 6160 tablet_service.cc:1514] Processing DeleteTablet for tablet de91942477d04c10bf563460db17057a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:55:31 UTC) from {username='slave'} at 127.0.0.1:49058
I20260619 14:55:31.248212 5898 tablet_service.cc:1514] Processing DeleteTablet for tablet de91942477d04c10bf563460db17057a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:55:31 UTC) from {username='slave'} at 127.0.0.1:36250
I20260619 14:55:31.248414 6267 tablet_replica.cc:330] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6: stopping tablet replica
I20260619 14:55:31.248677 6267 raft_consensus.cc:2229] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:55:31.248744 6266 tablet_replica.cc:330] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1: stopping tablet replica
I20260619 14:55:31.248965 6267 raft_consensus.cc:2258] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:55:31.248991 6266 raft_consensus.cc:2229] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:55:31.249011 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 5836
I20260619 14:55:31.249264 6266 raft_consensus.cc:2258] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:55:31.250917 6267 ts_tablet_manager.cc:1905] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:55:31.251932 6266 ts_tablet_manager.cc:1905] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:55:31.254210 6267 ts_tablet_manager.cc:1918] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:55:31.254300 6267 log.cc:1196] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/de91942477d04c10bf563460db17057a
I20260619 14:55:31.254370 6266 ts_tablet_manager.cc:1918] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:55:31.254446 6266 log.cc:1196] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880697016847-19593-0/minicluster-data/ts-1/wal/wals/de91942477d04c10bf563460db17057a
I20260619 14:55:31.254596 6267 ts_tablet_manager.cc:1939] T de91942477d04c10bf563460db17057a P 14c6d4f4f4e04d4c8acb0f4c97246aa6: Deleting consensus metadata
I20260619 14:55:31.254712 6266 ts_tablet_manager.cc:1939] T de91942477d04c10bf563460db17057a P c6ecccf94d44433b9704a9d82cb993c1: Deleting consensus metadata
I20260619 14:55:31.255652 5762 catalog_manager.cc:4841] TS c6ecccf94d44433b9704a9d82cb993c1 (127.19.34.66:46491): tablet de91942477d04c10bf563460db17057a (table test-table [id=20e305c474714135b5f23f0fe2821452]) successfully deleted
I20260619 14:55:31.256475 5760 catalog_manager.cc:4841] TS 14c6d4f4f4e04d4c8acb0f4c97246aa6 (127.19.34.67:39065): tablet de91942477d04c10bf563460db17057a (table test-table [id=20e305c474714135b5f23f0fe2821452]) successfully deleted
W20260619 14:55:31.257910 5763 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.34.65:34487 (error 108)
I20260619 14:55:31.258085 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 5967
I20260619 14:55:31.260118 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:31.259897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34487 (local address 127.0.0.1:36252)
0619 14:55:31.260007 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:31.260060 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.65:34487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 14:55:31.260308 5763 catalog_manager.cc:4584] TS e0a2b15a443e45b0961b08aca9267fe7 (127.19.34.65:34487): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de91942477d04c10bf563460db17057a: Network error: Client connection negotiation failed: client connection to 127.19.34.65:34487: connect: Connection refused (error 111)
I20260619 14:55:31.265087 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 6098
I20260619 14:55:31.269402 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 5745
2026-06-19T14:55:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1028 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:31 dist-test-slave-3qdh krb5kdc[6272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:31 dist-test-slave-3qdh krb5kdc[6272](info): set up 2 sockets
Jun 19 14:55:31 dist-test-slave-3qdh krb5kdc[6272](info): commencing operation
krb5kdc: starting...
W20260619 14:55:32.244149 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:32 dist-test-slave-3qdh krb5kdc[6272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:55:32.351280 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:36271
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:36271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:32.449127 6285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:32.449363 6285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:32.449426 6285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:32.452881 6285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:32.452968 6285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:32.453022 6285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:32.453058 6285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:32.453131 6285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:32.457777 6285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:36271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:36271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:32.458572 6285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:32.459390 6285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:32.466795 6290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:32.466871 6285 server_base.cc:1034] running on GCE node
W20260619 14:55:32.466780 6293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:32.466790 6291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:32.467519 6285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:32.468649 6285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:32.469849 6285 hybrid_clock.cc:648] HybridClock initialized: now 1781880932469825 us; error 41 us; skew 500 ppm
Jun 19 14:55:32 dist-test-slave-3qdh krb5kdc[6272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880932, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:32.472347 6285 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:55:32.473199 6285 webserver.cc:460] Webserver started at http://127.19.34.126:41103/ using document root <none> and password file <none>
I20260619 14:55:32.473718 6285 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:32.473829 6285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:32.474052 6285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:32.475808 6285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "f95deb63ebd449c1bef8a840273cf84a"
format_stamp: "Formatted at 2026-06-19 14:55:32 on dist-test-slave-3qdh"
server_key: "6c3f88666b798fe4aa7b6552e93867fc"
server_key_iv: "cad0e3d6f78fe21930341d05ac8a660e"
server_key_version: "encryptionkey@0"
I20260619 14:55:32.476390 6285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "f95deb63ebd449c1bef8a840273cf84a"
format_stamp: "Formatted at 2026-06-19 14:55:32 on dist-test-slave-3qdh"
server_key: "6c3f88666b798fe4aa7b6552e93867fc"
server_key_iv: "cad0e3d6f78fe21930341d05ac8a660e"
server_key_version: "encryptionkey@0"
I20260619 14:55:32.479506 6285 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:55:32.481663 6300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:32.482729 6285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:55:32.482887 6285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "f95deb63ebd449c1bef8a840273cf84a"
format_stamp: "Formatted at 2026-06-19 14:55:32 on dist-test-slave-3qdh"
server_key: "6c3f88666b798fe4aa7b6552e93867fc"
server_key_iv: "cad0e3d6f78fe21930341d05ac8a660e"
server_key_version: "encryptionkey@0"
I20260619 14:55:32.483001 6285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:32.503685 6285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:32.506227 6285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:32.506412 6285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:32.513037 6285 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:36271
I20260619 14:55:32.513053 6352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:36271 every 8 connection(s)
I20260619 14:55:32.513890 6285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:32.516182 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 6285
I20260619 14:55:32.516192 6353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:32.516280 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:32.516587 19593 external_mini_cluster.cc:1428] Setting key 4615a24c4153a5ce80514f78c3124dd6
I20260619 14:55:32.521533 6353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a: Bootstrap starting.
I20260619 14:55:32.523485 6353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:32.524219 6353 log.cc:824] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:55:32 dist-test-slave-3qdh krb5kdc[6272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:55:32.526081 6353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a: No bootstrap required, opened a new log
I20260619 14:55:32.529389 6356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:32.517836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.19.34.126:36271)
0619 14:55:32.518223 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:32.518236 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:32.518276 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:55:32.518996 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:32.519011 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:32.519281 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:32.519564 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:32.520280 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:32.521168 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:32.522008 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:32.522402 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:32.522548 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:32.525059 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:32.525064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:32.525075 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:32.525099 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:32.526661 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:32.527025 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:32.527030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:32.527035 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:32.527147 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:32.527381 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:32.527384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:32.527386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:32.527799 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:32.528048 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:32.528273 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 14:55:32.528576 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":132,"threads_started":1}
I20260619 14:55:32.530228 6353 raft_consensus.cc:348] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95deb63ebd449c1bef8a840273cf84a" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 36271 } }
I20260619 14:55:32.530435 6353 raft_consensus.cc:374] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:32.530485 6353 raft_consensus.cc:729] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f95deb63ebd449c1bef8a840273cf84a, State: Initialized, Role: FOLLOWER
I20260619 14:55:32.530864 6353 consensus_queue.cc:260] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95deb63ebd449c1bef8a840273cf84a" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 36271 } }
I20260619 14:55:32.531002 6353 raft_consensus.cc:388] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:32.531092 6353 raft_consensus.cc:482] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:32.531180 6353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:32.531840 6353 raft_consensus.cc:504] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95deb63ebd449c1bef8a840273cf84a" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 36271 } }
I20260619 14:55:32.532099 6353 leader_election.cc:302] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f95deb63ebd449c1bef8a840273cf84a; no voters:
I20260619 14:55:32.532413 6353 leader_election.cc:288] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:32.532495 6358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:32.532686 6358 raft_consensus.cc:686] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a [term 1 LEADER]: Becoming Leader. State: Replica: f95deb63ebd449c1bef8a840273cf84a, State: Running, Role: LEADER
I20260619 14:55:32.532974 6358 consensus_queue.cc:237] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95deb63ebd449c1bef8a840273cf84a" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 36271 } }
I20260619 14:55:32.533507 6353 sys_catalog.cc:564] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:32.534649 6360 sys_catalog.cc:455] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f95deb63ebd449c1bef8a840273cf84a. Latest consensus state: current_term: 1 leader_uuid: "f95deb63ebd449c1bef8a840273cf84a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95deb63ebd449c1bef8a840273cf84a" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 36271 } } }
I20260619 14:55:32.534689 6359 sys_catalog.cc:455] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f95deb63ebd449c1bef8a840273cf84a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95deb63ebd449c1bef8a840273cf84a" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 36271 } } }
I20260619 14:55:32.534775 6360 sys_catalog.cc:458] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:32.534811 6359 sys_catalog.cc:458] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:32.535163 6368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:32.537475 6368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:32.542094 6368 catalog_manager.cc:1347] Generated new cluster ID: 32b6280be5ee4d33851560729f697136
I20260619 14:55:32.542148 6368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:32.554044 6368 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:32.554761 6368 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:32.561120 6368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a: Generated new TSK 0
I20260619 14:55:32.561606 6368 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.34.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
Entry for principal kudu/127.19.34.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab.
I20260619 14:55:32.593065 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:36271
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:32.688549 6379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:32.688755 6379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:32.688836 6379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:32.692560 6379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:32.692605 6379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:32.692721 6379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:55:32.697480 6379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.65.keytab
--principal=kudu/127.19.34.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:36271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:32.698254 6379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:32.699049 6379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:32.707561 6384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:32.707559 6385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:32.707808 6387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:32.707798 6379 server_base.cc:1034] running on GCE node
I20260619 14:55:32.708200 6379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:32.708781 6379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:32.709941 6379 hybrid_clock.cc:648] HybridClock initialized: now 1781880932709903 us; error 50 us; skew 500 ppm
Jun 19 14:55:32 dist-test-slave-3qdh krb5kdc[6272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880932, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:32.712262 6379 init.cc:377] Logged in from keytab as kudu/127.19.34.65@KRBTEST.COM (short username kudu)
I20260619 14:55:32.713115 6379 webserver.cc:460] Webserver started at http://127.19.34.65:35793/ using document root <none> and password file <none>
I20260619 14:55:32.713613 6379 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:32.713670 6379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:32.713887 6379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:32.715528 6379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "4db2c49fe343496589479aac9b324692"
format_stamp: "Formatted at 2026-06-19 14:55:32 on dist-test-slave-3qdh"
server_key: "f875c48cb8035daed3a69cc55a0ee3cb"
server_key_iv: "21ac7a538ce5a3ef5e3cb6d7861780b6"
server_key_version: "encryptionkey@0"
I20260619 14:55:32.715997 6379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "4db2c49fe343496589479aac9b324692"
format_stamp: "Formatted at 2026-06-19 14:55:32 on dist-test-slave-3qdh"
server_key: "f875c48cb8035daed3a69cc55a0ee3cb"
server_key_iv: "21ac7a538ce5a3ef5e3cb6d7861780b6"
server_key_version: "encryptionkey@0"
I20260619 14:55:32.718912 6379 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:55:32.720887 6394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:32.721858 6379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:32.721961 6379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "4db2c49fe343496589479aac9b324692"
format_stamp: "Formatted at 2026-06-19 14:55:32 on dist-test-slave-3qdh"
server_key: "f875c48cb8035daed3a69cc55a0ee3cb"
server_key_iv: "21ac7a538ce5a3ef5e3cb6d7861780b6"
server_key_version: "encryptionkey@0"
I20260619 14:55:32.722050 6379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:32.734344 6379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:32.736958 6379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:32.737128 6379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:32.737589 6379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:55:32.738247 6379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:55:32.738323 6379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:32.738372 6379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:55:32.738422 6379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:32.745755 6379 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:36175
I20260619 14:55:32.745808 6507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:36175 every 8 connection(s)
I20260619 14:55:32.746511 6379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:55:32.747725 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 6379
I20260619 14:55:32.747888 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:55:32.748189 19593 external_mini_cluster.cc:1428] Setting key d25feea692297784f98cb6ef7024c9e1
Jun 19 14:55:32 dist-test-slave-3qdh krb5kdc[6272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880932, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.65@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.34.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.66@KRBTEST.COM" created.
I20260619 14:55:32.760254 6356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:32.748183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:59959 (local address 127.19.34.126:36271)
0619 14:55:32.748328 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:32.748332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:32.749211 (+ 879us) server_negotiation.cc:407] Connection header received
0619 14:55:32.750415 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:32.750418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:32.750465 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:32.750534 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:32.751823 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:32.752339 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:32.754035 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:32.754255 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:32.754324 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:32.756803 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:32.756811 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:32.756813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:32.756833 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:32.758308 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:32.758866 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:32.758870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:32.758872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:32.758965 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:32.759265 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:32.759270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:32.759275 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:32.759456 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:32.759525 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:32.759970 (+ 445us) server_negotiation.cc:299] Negotiation successful
0619 14:55:32.760060 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:55:32.761020 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:32.748605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:36271 (local address 127.19.34.65:59959)
0619 14:55:32.749027 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:32.749077 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:32.750236 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:32.750668 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:32.750680 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:32.751086 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:32.751677 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:32.751694 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:32.752446 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:32.752452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:32.753916 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:32.753929 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:32.754402 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:32.754408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:32.754618 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:32.755132 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:32.755155 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:32.756620 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:32.758511 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:32.758527 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:32.758535 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:32.758769 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:32.759051 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:32.759053 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:32.759054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:32.759174 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:32.759533 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:32.759537 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:32.759732 (+ 195us) client_negotiation.cc:769] Sending connection context
0619 14:55:32.759907 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 14:55:32.760298 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":93,"threads_started":1}
I20260619 14:55:32.762229 6508 heartbeater.cc:347] Connected to a master server at 127.19.34.126:36271
I20260619 14:55:32.762436 6508 heartbeater.cc:464] Registering TS with master...
I20260619 14:55:32.762858 6508 heartbeater.cc:511] Master 127.19.34.126:36271 requested a full tablet report, sending...
I20260619 14:55:32.764263 6317 ts_manager.cc:194] Registered new tserver with Master: 4db2c49fe343496589479aac9b324692 (127.19.34.65:36175)
I20260619 14:55:32.765336 6317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.65@KRBTEST.COM'} at 127.19.34.65:59959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
Entry for principal kudu/127.19.34.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab.
I20260619 14:55:32.775477 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:36271
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:32.877029 6513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:32.877231 6513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:32.877280 6513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:32.880754 6513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:32.880795 6513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:32.880843 6513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:55:32.885290 6513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.66.keytab
--principal=kudu/127.19.34.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:36271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:32.886042 6513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:32.886845 6513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:32.895028 6521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:32.895046 6518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:32.895054 6519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:32.895342 6513 server_base.cc:1034] running on GCE node
I20260619 14:55:32.895666 6513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:32.896142 6513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:32.897318 6513 hybrid_clock.cc:648] HybridClock initialized: now 1781880932897296 us; error 40 us; skew 500 ppm
Jun 19 14:55:32 dist-test-slave-3qdh krb5kdc[6272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880932, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:32.899681 6513 init.cc:377] Logged in from keytab as kudu/127.19.34.66@KRBTEST.COM (short username kudu)
I20260619 14:55:32.900471 6513 webserver.cc:460] Webserver started at http://127.19.34.66:34651/ using document root <none> and password file <none>
I20260619 14:55:32.900866 6513 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:32.900911 6513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:32.901032 6513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:32.902565 6513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "5fd412c727a141f0bee7ff9483e8b7c5"
format_stamp: "Formatted at 2026-06-19 14:55:32 on dist-test-slave-3qdh"
server_key: "5e086c1e0ea0512a4d4a85c5e203651f"
server_key_iv: "6fc623f04d012c7d59350fa86a924e0b"
server_key_version: "encryptionkey@0"
I20260619 14:55:32.903034 6513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "5fd412c727a141f0bee7ff9483e8b7c5"
format_stamp: "Formatted at 2026-06-19 14:55:32 on dist-test-slave-3qdh"
server_key: "5e086c1e0ea0512a4d4a85c5e203651f"
server_key_iv: "6fc623f04d012c7d59350fa86a924e0b"
server_key_version: "encryptionkey@0"
I20260619 14:55:32.906224 6513 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:55:32.908169 6528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:32.909288 6513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:32.909417 6513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "5fd412c727a141f0bee7ff9483e8b7c5"
format_stamp: "Formatted at 2026-06-19 14:55:32 on dist-test-slave-3qdh"
server_key: "5e086c1e0ea0512a4d4a85c5e203651f"
server_key_iv: "6fc623f04d012c7d59350fa86a924e0b"
server_key_version: "encryptionkey@0"
I20260619 14:55:32.909508 6513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:32.925896 6513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:32.928782 6513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:32.928972 6513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:32.929482 6513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:55:32.930167 6513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:55:32.930210 6513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:32.930303 6513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:55:32.930336 6513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:32.938441 6513 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:37013
I20260619 14:55:32.938477 6641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:37013 every 8 connection(s)
I20260619 14:55:32.939231 6513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:55:32.940150 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 6513
I20260619 14:55:32.940271 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:55:32.940605 19593 external_mini_cluster.cc:1428] Setting key 74224634248a7b006760afefc8294f35
Jun 19 14:55:32 dist-test-slave-3qdh krb5kdc[6272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880932, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.66@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.34.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.67@KRBTEST.COM" created.
I20260619 14:55:32.953260 6356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:32.940873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:35135 (local address 127.19.34.126:36271)
0619 14:55:32.940988 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:32.940992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:32.941860 (+ 868us) server_negotiation.cc:407] Connection header received
0619 14:55:32.943171 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:32.943177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:32.943236 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:32.943327 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:32.944710 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:32.945145 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:32.946861 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:32.947063 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:32.947139 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:32.949731 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:32.949734 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:32.949737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:32.949761 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:32.951138 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:32.951646 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:32.951658 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:32.951660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:32.951734 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:32.952138 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:32.952141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:32.952142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:32.952361 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:32.952466 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:32.952987 (+ 521us) server_negotiation.cc:299] Negotiation successful
0619 14:55:32.953098 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:55:32.953923 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:32.941284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:36271 (local address 127.19.34.66:35135)
0619 14:55:32.941702 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:32.941749 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:32.942956 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:32.943459 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:32.943465 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:32.943985 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:32.944564 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:32.944581 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:32.945292 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:32.945298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:32.946734 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:32.946747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:32.947212 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:32.947218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:32.947440 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:32.948005 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:32.948026 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:32.949573 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:32.951280 (+ 1707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:32.951285 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:32.951297 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:32.951537 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:32.951828 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:32.951832 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:32.951834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:32.951985 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:32.952464 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:32.952488 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:32.952713 (+ 225us) client_negotiation.cc:769] Sending connection context
0619 14:55:32.952942 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 14:55:32.953223 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":98,"threads_started":1}
I20260619 14:55:32.954957 6642 heartbeater.cc:347] Connected to a master server at 127.19.34.126:36271
I20260619 14:55:32.955173 6642 heartbeater.cc:464] Registering TS with master...
I20260619 14:55:32.955596 6642 heartbeater.cc:511] Master 127.19.34.126:36271 requested a full tablet report, sending...
I20260619 14:55:32.956605 6317 ts_manager.cc:194] Registered new tserver with Master: 5fd412c727a141f0bee7ff9483e8b7c5 (127.19.34.66:37013)
I20260619 14:55:32.957106 6317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.66@KRBTEST.COM'} at 127.19.34.66:35135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
Entry for principal kudu/127.19.34.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab.
I20260619 14:55:32.967170 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:36271
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:33.064757 6647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:33.064968 6647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:33.065026 6647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:33.068463 6647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:33.068535 6647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:33.068619 6647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:55:33.073379 6647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.67.keytab
--principal=kudu/127.19.34.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:36271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:33.074244 6647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:33.075016 6647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:33.082772 6652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:33.082772 6655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:33.082792 6653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:33.082960 6647 server_base.cc:1034] running on GCE node
I20260619 14:55:33.083356 6647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:33.083958 6647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:33.085098 6647 hybrid_clock.cc:648] HybridClock initialized: now 1781880933085079 us; error 49 us; skew 500 ppm
Jun 19 14:55:33 dist-test-slave-3qdh krb5kdc[6272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880933, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:33.087679 6647 init.cc:377] Logged in from keytab as kudu/127.19.34.67@KRBTEST.COM (short username kudu)
I20260619 14:55:33.088495 6647 webserver.cc:460] Webserver started at http://127.19.34.67:37811/ using document root <none> and password file <none>
I20260619 14:55:33.088975 6647 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:33.089032 6647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:33.089242 6647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:33.090833 6647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "0665286525e746cda3c8f8a7df1064b3"
format_stamp: "Formatted at 2026-06-19 14:55:33 on dist-test-slave-3qdh"
server_key: "1e0bbaa6517756006cfbf63ec975f99b"
server_key_iv: "b5803d6d520576690ec0f3f9745a401c"
server_key_version: "encryptionkey@0"
I20260619 14:55:33.091301 6647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "0665286525e746cda3c8f8a7df1064b3"
format_stamp: "Formatted at 2026-06-19 14:55:33 on dist-test-slave-3qdh"
server_key: "1e0bbaa6517756006cfbf63ec975f99b"
server_key_iv: "b5803d6d520576690ec0f3f9745a401c"
server_key_version: "encryptionkey@0"
I20260619 14:55:33.094792 6647 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:55:33.096688 6662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:33.097710 6647 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260619 14:55:33.097812 6647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "0665286525e746cda3c8f8a7df1064b3"
format_stamp: "Formatted at 2026-06-19 14:55:33 on dist-test-slave-3qdh"
server_key: "1e0bbaa6517756006cfbf63ec975f99b"
server_key_iv: "b5803d6d520576690ec0f3f9745a401c"
server_key_version: "encryptionkey@0"
I20260619 14:55:33.097899 6647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:33.111660 6647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:33.114658 6647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:33.114818 6647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:33.115288 6647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:55:33.115932 6647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:55:33.115970 6647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:33.116029 6647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:55:33.116062 6647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:33.124837 6647 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:35023
I20260619 14:55:33.124938 6775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:35023 every 8 connection(s)
I20260619 14:55:33.125614 6647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:55:33.132185 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 6647
I20260619 14:55:33.132336 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:55:33.132616 19593 external_mini_cluster.cc:1428] Setting key 3421908c7b5d7c2a46d1dc14e35fd3b1
Jun 19 14:55:33 dist-test-slave-3qdh krb5kdc[6272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880933, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.67@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:55:33.137728 6356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:33.127174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:36865 (local address 127.19.34.126:36271)
0619 14:55:33.127309 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:33.127314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:33.128054 (+ 740us) server_negotiation.cc:407] Connection header received
0619 14:55:33.129218 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:33.129221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:33.129285 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:33.129363 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:33.130263 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.130777 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.131728 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.131949 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.132080 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:33.134543 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:33.134546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:33.134549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:33.134573 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:33.135958 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:33.136411 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:33.136414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:33.136416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:33.136494 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:33.136799 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:33.136801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:33.136803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:33.136957 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:33.137051 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:33.137436 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 14:55:33.137556 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:55:33.138295 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:33.127508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:36271 (local address 127.19.34.67:36865)
0619 14:55:33.127914 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:33.127949 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:33.129016 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:33.129505 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:33.129510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:33.129775 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:33.130140 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:33.130151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.130881 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.130883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:33.131621 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:33.131629 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.132082 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.132084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:33.132209 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:33.132784 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:33.132801 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:33.134380 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:33.136063 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:33.136068 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:33.136078 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:33.136292 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:33.136583 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:33.136585 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:33.136587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:33.136706 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:33.137050 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:33.137055 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:33.137241 (+ 186us) client_negotiation.cc:769] Sending connection context
0619 14:55:33.137390 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 14:55:33.137687 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":92,"threads_started":1}
I20260619 14:55:33.139416 6776 heartbeater.cc:347] Connected to a master server at 127.19.34.126:36271
I20260619 14:55:33.139588 6776 heartbeater.cc:464] Registering TS with master...
I20260619 14:55:33.139928 6776 heartbeater.cc:511] Master 127.19.34.126:36271 requested a full tablet report, sending...
I20260619 14:55:33.140823 6317 ts_manager.cc:194] Registered new tserver with Master: 0665286525e746cda3c8f8a7df1064b3 (127.19.34.67:35023)
I20260619 14:55:33.141332 6317 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.34.67@KRBTEST.COM'} at 127.19.34.67:36865
I20260619 14:55:33.146304 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:55:33.154009 6356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:33.148074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.19.34.126:36271)
0619 14:55:33.148192 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:33.148195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:33.148206 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:55:33.148343 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:33.148346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:33.148385 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:33.148456 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:33.149088 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.149498 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.150214 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.150387 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.150450 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:33.151500 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:33.151502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:33.151505 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:33.151522 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:33.152672 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:33.153065 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:33.153067 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:33.153068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:33.153132 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:33.153356 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:33.153358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:33.153359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:33.153530 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:33.153624 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:33.153821 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 14:55:33.153911 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:55:33.156205 6317 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:55:33.158735 6317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:55:33.170928 6788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:33.165792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:37013 (local address 127.0.0.1:52360)
0619 14:55:33.166126 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:33.166138 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:33.166212 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:33.166874 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:33.166879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:33.166901 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:33.166992 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:33.166998 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.168198 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.168203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:33.169476 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:33.169485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.170465 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.170469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:33.170562 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:33.170606 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 14:55:33.170710 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 14:55:33.170807 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":81,"threads_started":1}
I20260619 14:55:33.171010 6789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:33.165926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52360 (local address 127.19.34.66:37013)
0619 14:55:33.166438 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:33.166448 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:33.166468 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:55:33.166535 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:33.166543 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:33.166731 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:33.167128 (+ 397us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:33.167279 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.168068 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.169614 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.170346 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.170443 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:33.170564 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:33.170755 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 14:55:33.170844 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":104,"threads_started":1}
I20260619 14:55:33.171337 6791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:33.166483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:35023 (local address 127.0.0.1:57036)
0619 14:55:33.167124 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:33.167135 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:33.167215 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:33.167596 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:33.167599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:33.167614 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:33.167678 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:33.167683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.168957 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.168963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:33.169962 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:33.169970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.170886 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.170889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:33.171004 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:33.171050 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 14:55:33.171142 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 14:55:33.171203 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":3,"thread_start_us":77,"threads_started":1}
I20260619 14:55:33.171554 6787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:33.165585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:36175 (local address 127.0.0.1:42572)
0619 14:55:33.165853 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:33.165897 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:33.165989 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:33.166966 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:33.166969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:33.166985 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:33.167105 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:33.167111 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.168692 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.168695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:33.169810 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:33.169818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.170817 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.170820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:33.170911 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:33.170931 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:55:33.171000 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 14:55:33.171438 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":353,"thread_start_us":70,"threads_started":1}
I20260619 14:55:33.171730 6792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:33.166599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.19.34.67:35023)
0619 14:55:33.166926 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:33.166936 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:33.167269 (+ 333us) server_negotiation.cc:407] Connection header received
0619 14:55:33.167327 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:33.167331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:33.167463 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:33.167585 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:33.167793 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.168835 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.170074 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.170647 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.170729 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:33.170833 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:33.171510 (+ 677us) server_negotiation.cc:299] Negotiation successful
0619 14:55:33.171591 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":113,"threads_started":1}
I20260619 14:55:33.172230 6790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:33.166133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.19.34.65:36175)
0619 14:55:33.166547 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:33.166560 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:33.166582 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:55:33.166660 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:33.166664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:33.166828 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:33.167236 (+ 408us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:33.167390 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.168538 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.169927 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.170668 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.171831 (+ 1163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:33.171961 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:33.172009 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 14:55:33.172090 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":113,"threads_started":1}
I20260619 14:55:33.173878 6710 tablet_service.cc:1467] Processing CreateTablet for tablet 128e409a803c479ca83b2c99a03c8f11 (DEFAULT_TABLE table=test-table [id=2f6fd97ad2394432aaf9f267d9ce880c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:55:33.173895 6576 tablet_service.cc:1467] Processing CreateTablet for tablet 128e409a803c479ca83b2c99a03c8f11 (DEFAULT_TABLE table=test-table [id=2f6fd97ad2394432aaf9f267d9ce880c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:55:33.174610 6710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128e409a803c479ca83b2c99a03c8f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:33.174767 6576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128e409a803c479ca83b2c99a03c8f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:33.174566 6442 tablet_service.cc:1467] Processing CreateTablet for tablet 128e409a803c479ca83b2c99a03c8f11 (DEFAULT_TABLE table=test-table [id=2f6fd97ad2394432aaf9f267d9ce880c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:55:33.175325 6442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128e409a803c479ca83b2c99a03c8f11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:33.179777 6793 tablet_bootstrap.cc:492] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3: Bootstrap starting.
I20260619 14:55:33.181720 6795 tablet_bootstrap.cc:492] T 128e409a803c479ca83b2c99a03c8f11 P 5fd412c727a141f0bee7ff9483e8b7c5: Bootstrap starting.
I20260619 14:55:33.181921 6793 tablet_bootstrap.cc:654] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:33.182960 6793 log.cc:824] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:33.183444 6795 tablet_bootstrap.cc:654] T 128e409a803c479ca83b2c99a03c8f11 P 5fd412c727a141f0bee7ff9483e8b7c5: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:33.183655 6794 tablet_bootstrap.cc:492] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692: Bootstrap starting.
I20260619 14:55:33.184520 6795 log.cc:824] T 128e409a803c479ca83b2c99a03c8f11 P 5fd412c727a141f0bee7ff9483e8b7c5: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:33.184924 6793 tablet_bootstrap.cc:492] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3: No bootstrap required, opened a new log
I20260619 14:55:33.185096 6793 ts_tablet_manager.cc:1397] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20260619 14:55:33.186024 6794 tablet_bootstrap.cc:654] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:33.186215 6795 tablet_bootstrap.cc:492] T 128e409a803c479ca83b2c99a03c8f11 P 5fd412c727a141f0bee7ff9483e8b7c5: No bootstrap required, opened a new log
I20260619 14:55:33.186414 6795 ts_tablet_manager.cc:1397] T 128e409a803c479ca83b2c99a03c8f11 P 5fd412c727a141f0bee7ff9483e8b7c5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:55:33.186786 6794 log.cc:824] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:33.188652 6794 tablet_bootstrap.cc:492] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692: No bootstrap required, opened a new log
I20260619 14:55:33.188838 6794 ts_tablet_manager.cc:1397] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:55:33.189049 6793 raft_consensus.cc:348] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db2c49fe343496589479aac9b324692" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 36175 } } peers { permanent_uuid: "5fd412c727a141f0bee7ff9483e8b7c5" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 37013 } } peers { permanent_uuid: "0665286525e746cda3c8f8a7df1064b3" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35023 } }
I20260619 14:55:33.189322 6793 raft_consensus.cc:374] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:33.189376 6793 raft_consensus.cc:729] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0665286525e746cda3c8f8a7df1064b3, State: Initialized, Role: FOLLOWER
I20260619 14:55:33.189608 6795 raft_consensus.cc:348] T 128e409a803c479ca83b2c99a03c8f11 P 5fd412c727a141f0bee7ff9483e8b7c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db2c49fe343496589479aac9b324692" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 36175 } } peers { permanent_uuid: "5fd412c727a141f0bee7ff9483e8b7c5" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 37013 } } peers { permanent_uuid: "0665286525e746cda3c8f8a7df1064b3" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35023 } }
I20260619 14:55:33.189818 6795 raft_consensus.cc:374] T 128e409a803c479ca83b2c99a03c8f11 P 5fd412c727a141f0bee7ff9483e8b7c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:33.189759 6793 consensus_queue.cc:260] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db2c49fe343496589479aac9b324692" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 36175 } } peers { permanent_uuid: "5fd412c727a141f0bee7ff9483e8b7c5" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 37013 } } peers { permanent_uuid: "0665286525e746cda3c8f8a7df1064b3" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35023 } }
I20260619 14:55:33.189869 6795 raft_consensus.cc:729] T 128e409a803c479ca83b2c99a03c8f11 P 5fd412c727a141f0bee7ff9483e8b7c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd412c727a141f0bee7ff9483e8b7c5, State: Initialized, Role: FOLLOWER
I20260619 14:55:33.190322 6795 consensus_queue.cc:260] T 128e409a803c479ca83b2c99a03c8f11 P 5fd412c727a141f0bee7ff9483e8b7c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db2c49fe343496589479aac9b324692" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 36175 } } peers { permanent_uuid: "5fd412c727a141f0bee7ff9483e8b7c5" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 37013 } } peers { permanent_uuid: "0665286525e746cda3c8f8a7df1064b3" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35023 } }
I20260619 14:55:33.190586 6793 ts_tablet_manager.cc:1428] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260619 14:55:33.190893 6642 heartbeater.cc:503] Master 127.19.34.126:36271 was elected leader, sending a full tablet report...
I20260619 14:55:33.190897 6776 heartbeater.cc:503] Master 127.19.34.126:36271 was elected leader, sending a full tablet report...
I20260619 14:55:33.191116 6795 ts_tablet_manager.cc:1428] T 128e409a803c479ca83b2c99a03c8f11 P 5fd412c727a141f0bee7ff9483e8b7c5: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:55:33.192704 6794 raft_consensus.cc:348] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db2c49fe343496589479aac9b324692" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 36175 } } peers { permanent_uuid: "5fd412c727a141f0bee7ff9483e8b7c5" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 37013 } } peers { permanent_uuid: "0665286525e746cda3c8f8a7df1064b3" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35023 } }
I20260619 14:55:33.193001 6794 raft_consensus.cc:374] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:33.193070 6794 raft_consensus.cc:729] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db2c49fe343496589479aac9b324692, State: Initialized, Role: FOLLOWER
I20260619 14:55:33.193472 6794 consensus_queue.cc:260] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db2c49fe343496589479aac9b324692" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 36175 } } peers { permanent_uuid: "5fd412c727a141f0bee7ff9483e8b7c5" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 37013 } } peers { permanent_uuid: "0665286525e746cda3c8f8a7df1064b3" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35023 } }
I20260619 14:55:33.194281 6508 heartbeater.cc:503] Master 127.19.34.126:36271 was elected leader, sending a full tablet report...
I20260619 14:55:33.194437 6794 ts_tablet_manager.cc:1428] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260619 14:55:33.247502 6509 tablet.cc:2369] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:55:33.375280 6801 raft_consensus.cc:482] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:55:33.375547 6801 raft_consensus.cc:504] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db2c49fe343496589479aac9b324692" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 36175 } } peers { permanent_uuid: "5fd412c727a141f0bee7ff9483e8b7c5" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 37013 } } peers { permanent_uuid: "0665286525e746cda3c8f8a7df1064b3" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35023 } }
W20260619 14:55:33.376489 6777 tablet.cc:2369] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:55:33.376698 6801 leader_election.cc:288] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fd412c727a141f0bee7ff9483e8b7c5 (127.19.34.66:37013), 0665286525e746cda3c8f8a7df1064b3 (127.19.34.67:35023)
I20260619 14:55:33.380702 6789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:33.377179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:33695 (local address 127.19.34.66:37013)
0619 14:55:33.377377 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:33.377381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:33.377408 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:55:33.377470 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:33.377474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:33.377525 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:33.377608 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:33.377954 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.378548 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.379665 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.380121 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.380266 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:33.380344 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:33.380505 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 14:55:33.380572 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260619 14:55:33.380712 6802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:33.376889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:37013 (local address 127.19.34.65:33695)
0619 14:55:33.377163 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:33.377177 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:33.377277 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:33.377627 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:33.377630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:33.377648 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:33.377794 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:33.377802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.378712 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.378716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:33.379529 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:33.379538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.380259 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.380263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:33.380384 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:33.380410 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:55:33.380465 (+ 55us) client_negotiation.cc:240] Negotiation successful
0619 14:55:33.380538 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":1,"thread_start_us":82,"threads_started":1}
I20260619 14:55:33.380754 6792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:33.377042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:38449 (local address 127.19.34.67:35023)
0619 14:55:33.377157 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:33.377161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:33.377501 (+ 340us) server_negotiation.cc:407] Connection header received
0619 14:55:33.377627 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:33.377630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:33.377672 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:33.377742 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:33.378139 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.378605 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.379741 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.380122 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.380276 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:33.380356 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:33.380576 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 14:55:33.380637 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:55:33.380926 6803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:33.376907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:35023 (local address 127.19.34.65:38449)
0619 14:55:33.377395 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:33.377408 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:33.377508 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:33.377833 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:33.377838 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:33.377863 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:33.378019 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:33.378026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.378756 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.378758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:33.379558 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:33.379566 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.380332 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.380336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:33.380427 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:33.380443 (+ 16us) client_negotiation.cc:769] Sending connection context
0619 14:55:33.380539 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 14:55:33.380605 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":22,"thread_start_us":71,"threads_started":1}
I20260619 14:55:33.381335 6596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128e409a803c479ca83b2c99a03c8f11" candidate_uuid: "4db2c49fe343496589479aac9b324692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd412c727a141f0bee7ff9483e8b7c5" is_pre_election: true
I20260619 14:55:33.381431 6730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128e409a803c479ca83b2c99a03c8f11" candidate_uuid: "4db2c49fe343496589479aac9b324692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0665286525e746cda3c8f8a7df1064b3" is_pre_election: true
I20260619 14:55:33.381628 6730 raft_consensus.cc:2454] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4db2c49fe343496589479aac9b324692 in term 0.
I20260619 14:55:33.381626 6596 raft_consensus.cc:2454] T 128e409a803c479ca83b2c99a03c8f11 P 5fd412c727a141f0bee7ff9483e8b7c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4db2c49fe343496589479aac9b324692 in term 0.
I20260619 14:55:33.382099 6396 leader_election.cc:302] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0665286525e746cda3c8f8a7df1064b3, 4db2c49fe343496589479aac9b324692, 5fd412c727a141f0bee7ff9483e8b7c5; no voters:
I20260619 14:55:33.382361 6801 raft_consensus.cc:2781] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:55:33.382436 6801 raft_consensus.cc:482] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:55:33.382465 6801 raft_consensus.cc:3037] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:33.383495 6801 raft_consensus.cc:504] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db2c49fe343496589479aac9b324692" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 36175 } } peers { permanent_uuid: "5fd412c727a141f0bee7ff9483e8b7c5" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 37013 } } peers { permanent_uuid: "0665286525e746cda3c8f8a7df1064b3" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35023 } }
I20260619 14:55:33.383826 6801 leader_election.cc:288] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692 [CANDIDATE]: Term 1 election: Requested vote from peers 5fd412c727a141f0bee7ff9483e8b7c5 (127.19.34.66:37013), 0665286525e746cda3c8f8a7df1064b3 (127.19.34.67:35023)
I20260619 14:55:33.384198 6730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128e409a803c479ca83b2c99a03c8f11" candidate_uuid: "4db2c49fe343496589479aac9b324692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0665286525e746cda3c8f8a7df1064b3"
I20260619 14:55:33.384198 6596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "128e409a803c479ca83b2c99a03c8f11" candidate_uuid: "4db2c49fe343496589479aac9b324692" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd412c727a141f0bee7ff9483e8b7c5"
I20260619 14:55:33.384335 6596 raft_consensus.cc:3037] T 128e409a803c479ca83b2c99a03c8f11 P 5fd412c727a141f0bee7ff9483e8b7c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:33.384335 6730 raft_consensus.cc:3037] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:33.385550 6596 raft_consensus.cc:2454] T 128e409a803c479ca83b2c99a03c8f11 P 5fd412c727a141f0bee7ff9483e8b7c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4db2c49fe343496589479aac9b324692 in term 1.
I20260619 14:55:33.385550 6730 raft_consensus.cc:2454] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4db2c49fe343496589479aac9b324692 in term 1.
I20260619 14:55:33.385802 6396 leader_election.cc:302] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0665286525e746cda3c8f8a7df1064b3, 4db2c49fe343496589479aac9b324692; no voters:
I20260619 14:55:33.385951 6801 raft_consensus.cc:2781] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:33.386217 6801 raft_consensus.cc:686] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692 [term 1 LEADER]: Becoming Leader. State: Replica: 4db2c49fe343496589479aac9b324692, State: Running, Role: LEADER
I20260619 14:55:33.386576 6801 consensus_queue.cc:237] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db2c49fe343496589479aac9b324692" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 36175 } } peers { permanent_uuid: "5fd412c727a141f0bee7ff9483e8b7c5" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 37013 } } peers { permanent_uuid: "0665286525e746cda3c8f8a7df1064b3" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35023 } }
I20260619 14:55:33.389684 6315 catalog_manager.cc:5495] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4db2c49fe343496589479aac9b324692 (127.19.34.65). New cstate: current_term: 1 leader_uuid: "4db2c49fe343496589479aac9b324692" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db2c49fe343496589479aac9b324692" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 36175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fd412c727a141f0bee7ff9483e8b7c5" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 37013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0665286525e746cda3c8f8a7df1064b3" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35023 } health_report { overall_health: UNKNOWN } } }
I20260619 14:55:33.401129 6790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:33.397932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42576 (local address 127.19.34.65:36175)
0619 14:55:33.398047 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:33.398053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:33.398111 (+ 58us) server_negotiation.cc:407] Connection header received
0619 14:55:33.398303 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:33.398306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:33.398351 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:33.398448 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:33.398664 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.399086 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.399849 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.400054 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.400152 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:33.400441 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:33.400760 (+ 319us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:33.400927 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:33.400974 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 14:55:33.401015 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:55:33.406076 6596 raft_consensus.cc:1261] T 128e409a803c479ca83b2c99a03c8f11 P 5fd412c727a141f0bee7ff9483e8b7c5 [term 1 FOLLOWER]: Refusing update from remote peer 4db2c49fe343496589479aac9b324692: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:55:33.406073 6730 raft_consensus.cc:1261] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3 [term 1 FOLLOWER]: Refusing update from remote peer 4db2c49fe343496589479aac9b324692: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:55:33.406644 6804 consensus_queue.cc:1035] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fd412c727a141f0bee7ff9483e8b7c5" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 37013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:55:33.406813 6801 consensus_queue.cc:1035] T 128e409a803c479ca83b2c99a03c8f11 P 4db2c49fe343496589479aac9b324692 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0665286525e746cda3c8f8a7df1064b3" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 35023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:55:33.414835 6809 mvcc.cc:205] Tried to move back new op lower bound from 7298584303225921536 to 7298584303154073600. Current Snapshot: MvccSnapshot[applied={T|T < 7298584303225921536}]
I20260619 14:55:33.422572 6317 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:55:33.422703 6317 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:55:33.424942 6317 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f95deb63ebd449c1bef8a840273cf84a: Sending DeleteTablet for 3 replicas of tablet 128e409a803c479ca83b2c99a03c8f11
I20260619 14:55:33.425594 6576 tablet_service.cc:1514] Processing DeleteTablet for tablet 128e409a803c479ca83b2c99a03c8f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:55:33 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:52360
I20260619 14:55:33.425594 6442 tablet_service.cc:1514] Processing DeleteTablet for tablet 128e409a803c479ca83b2c99a03c8f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:55:33 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:42572
I20260619 14:55:33.425988 6818 tablet_replica.cc:330] T 128e409a803c479ca83b2c99a03c8f11 P 5fd412c727a141f0bee7ff9483e8b7c5: stopping tablet replica
I20260619 14:55:33.426113 6710 tablet_service.cc:1514] Processing DeleteTablet for tablet 128e409a803c479ca83b2c99a03c8f11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:55:33 UTC) from {username='kudu', principal='kudu/127.19.34.126@KRBTEST.COM'} at 127.0.0.1:57036
I20260619 14:55:33.426350 6818 raft_consensus.cc:2229] T 128e409a803c479ca83b2c99a03c8f11 P 5fd412c727a141f0bee7ff9483e8b7c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:55:33.426631 6818 raft_consensus.cc:2258] T 128e409a803c479ca83b2c99a03c8f11 P 5fd412c727a141f0bee7ff9483e8b7c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:55:33.426713 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 6379
I20260619 14:55:33.428468 6818 ts_tablet_manager.cc:1905] T 128e409a803c479ca83b2c99a03c8f11 P 5fd412c727a141f0bee7ff9483e8b7c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:55:33.428994 6820 tablet_replica.cc:330] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3: stopping tablet replica
I20260619 14:55:33.429260 6820 raft_consensus.cc:2229] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:55:33.429481 6820 raft_consensus.cc:2258] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:55:33.430526 6820 ts_tablet_manager.cc:1905] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:55:33.431304 6818 ts_tablet_manager.cc:1918] T 128e409a803c479ca83b2c99a03c8f11 P 5fd412c727a141f0bee7ff9483e8b7c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:55:33.431399 6818 log.cc:1196] T 128e409a803c479ca83b2c99a03c8f11 P 5fd412c727a141f0bee7ff9483e8b7c5: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-1/wal/wals/128e409a803c479ca83b2c99a03c8f11
I20260619 14:55:33.431720 6818 ts_tablet_manager.cc:1939] T 128e409a803c479ca83b2c99a03c8f11 P 5fd412c727a141f0bee7ff9483e8b7c5: Deleting consensus metadata
I20260619 14:55:33.432688 6820 ts_tablet_manager.cc:1918] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:55:33.432830 6820 log.cc:1196] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/128e409a803c479ca83b2c99a03c8f11
I20260619 14:55:33.432891 6303 catalog_manager.cc:4841] TS 5fd412c727a141f0bee7ff9483e8b7c5 (127.19.34.66:37013): tablet 128e409a803c479ca83b2c99a03c8f11 (table test-table [id=2f6fd97ad2394432aaf9f267d9ce880c]) successfully deleted
I20260619 14:55:33.433187 6820 ts_tablet_manager.cc:1939] T 128e409a803c479ca83b2c99a03c8f11 P 0665286525e746cda3c8f8a7df1064b3: Deleting consensus metadata
I20260619 14:55:33.434145 6302 catalog_manager.cc:4841] TS 0665286525e746cda3c8f8a7df1064b3 (127.19.34.67:35023): tablet 128e409a803c479ca83b2c99a03c8f11 (table test-table [id=2f6fd97ad2394432aaf9f267d9ce880c]) successfully deleted
W20260619 14:55:33.434742 6304 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.34.65:36175 (error 108)
I20260619 14:55:33.434957 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 6513
I20260619 14:55:33.436987 6787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:33.436760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:36175 (local address 127.0.0.1:42588)
0619 14:55:33.436854 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:33.436917 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.65:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 14:55:33.437152 6304 catalog_manager.cc:4584] TS 4db2c49fe343496589479aac9b324692 (127.19.34.65:36175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 128e409a803c479ca83b2c99a03c8f11: Network error: Client connection negotiation failed: client connection to 127.19.34.65:36175: connect: Connection refused (error 111)
I20260619 14:55:33.440434 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 6647
I20260619 14:55:33.445272 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 6285
2026-06-19T14:55:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2177 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T14:55:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:33Z Disabled control of system clock
I20260619 14:55:33.473466 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:40949
--webserver_interface=127.19.34.126
--webserver_port=0
--builtin_ntp_servers=127.19.34.84:39629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:40949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:55:33.569226 6825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:33.569427 6825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:33.569459 6825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:33.573465 6825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:33.573542 6825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:33.573565 6825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:33.573586 6825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:33.573616 6825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:33.577797 6825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:39629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:40949
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:40949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:33.578493 6825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:33.579519 6825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:33.586722 6831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:33.586824 6830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:33.586730 6833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:33.586892 6825 server_base.cc:1034] running on GCE node
I20260619 14:55:33.587508 6825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:33.588487 6825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:33.589674 6825 hybrid_clock.cc:648] HybridClock initialized: now 1781880933589650 us; error 36 us; skew 500 ppm
I20260619 14:55:33.591365 6825 webserver.cc:460] Webserver started at http://127.19.34.126:33305/ using document root <none> and password file <none>
I20260619 14:55:33.591935 6825 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:33.592038 6825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:33.592252 6825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:33.594144 6825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "97a9680b4cc64a43a645e2c234b4d40d"
format_stamp: "Formatted at 2026-06-19 14:55:33 on dist-test-slave-3qdh"
server_key: "1f2fc39ba4c3312e22b60d4b53c32273"
server_key_iv: "3944d865a7896de2b92e205ac78fa0d4"
server_key_version: "encryptionkey@0"
I20260619 14:55:33.594672 6825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "97a9680b4cc64a43a645e2c234b4d40d"
format_stamp: "Formatted at 2026-06-19 14:55:33 on dist-test-slave-3qdh"
server_key: "1f2fc39ba4c3312e22b60d4b53c32273"
server_key_iv: "3944d865a7896de2b92e205ac78fa0d4"
server_key_version: "encryptionkey@0"
I20260619 14:55:33.598073 6825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:55:33.600035 6839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:33.601329 6825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:33.601473 6825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "97a9680b4cc64a43a645e2c234b4d40d"
format_stamp: "Formatted at 2026-06-19 14:55:33 on dist-test-slave-3qdh"
server_key: "1f2fc39ba4c3312e22b60d4b53c32273"
server_key_iv: "3944d865a7896de2b92e205ac78fa0d4"
server_key_version: "encryptionkey@0"
I20260619 14:55:33.601583 6825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:33.629590 6825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:33.630121 6825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:33.630291 6825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:33.636467 6825 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:40949
I20260619 14:55:33.636487 6891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:40949 every 8 connection(s)
I20260619 14:55:33.637183 6825 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:33.637867 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 6825
I20260619 14:55:33.637995 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:33.638263 19593 external_mini_cluster.cc:1428] Setting key 3505e9b18ee91b04089c276179e90859
I20260619 14:55:33.639932 6892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:33.645346 6892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d: Bootstrap starting.
I20260619 14:55:33.646246 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:33.639337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48572 (local address 127.19.34.126:40949)
0619 14:55:33.639866 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:33.639878 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:33.639907 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:55:33.640540 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:33.640554 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:33.640871 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:33.641103 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:33.641441 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.642240 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.643391 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.643817 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.643924 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:33.644813 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:33.644817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:33.644827 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:33.645138 (+ 311us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:33.645186 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:33.645203 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:33.645284 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:33.645483 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 14:55:33.645725 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":132,"threads_started":1}
I20260619 14:55:33.647203 6892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:33.647895 6892 log.cc:824] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:33.649483 6892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d: No bootstrap required, opened a new log
I20260619 14:55:33.652112 6892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a9680b4cc64a43a645e2c234b4d40d" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 40949 } }
I20260619 14:55:33.652352 6892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:33.652410 6892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97a9680b4cc64a43a645e2c234b4d40d, State: Initialized, Role: FOLLOWER
I20260619 14:55:33.652794 6892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a9680b4cc64a43a645e2c234b4d40d" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 40949 } }
I20260619 14:55:33.652933 6892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:33.652997 6892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:33.653084 6892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:33.654028 6892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a9680b4cc64a43a645e2c234b4d40d" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 40949 } }
I20260619 14:55:33.654336 6892 leader_election.cc:302] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97a9680b4cc64a43a645e2c234b4d40d; no voters:
I20260619 14:55:33.654597 6892 leader_election.cc:288] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:33.654699 6897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:33.654925 6897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d [term 1 LEADER]: Becoming Leader. State: Replica: 97a9680b4cc64a43a645e2c234b4d40d, State: Running, Role: LEADER
I20260619 14:55:33.655208 6897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a9680b4cc64a43a645e2c234b4d40d" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 40949 } }
I20260619 14:55:33.655622 6892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:33.656601 6899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97a9680b4cc64a43a645e2c234b4d40d. Latest consensus state: current_term: 1 leader_uuid: "97a9680b4cc64a43a645e2c234b4d40d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a9680b4cc64a43a645e2c234b4d40d" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 40949 } } }
I20260619 14:55:33.656703 6899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:33.657042 6906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:33.657081 6898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97a9680b4cc64a43a645e2c234b4d40d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a9680b4cc64a43a645e2c234b4d40d" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 40949 } } }
I20260619 14:55:33.657287 6898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:33.659363 6906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:33.663738 6906 catalog_manager.cc:1347] Generated new cluster ID: 7d2696f768e94e40b026fe2bd5fcd500
I20260619 14:55:33.663797 6906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:33.671344 6906 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:33.672290 6906 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:33.676690 6906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d: Generated new TSK 0
I20260619 14:55:33.677248 6906 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:33.682771 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.65:0
--local_ip_for_outbound_sockets=127.19.34.65
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:40949
--builtin_ntp_servers=127.19.34.84:39629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:55:33.781509 6916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:33.781747 6916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:33.781803 6916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:33.785756 6916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:33.785828 6916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:33.785907 6916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.65
I20260619 14:55:33.790516 6916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:39629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.34.65
--webserver_port=0
--tserver_master_addrs=127.19.34.126:40949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.65
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:33.791436 6916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:33.792672 6916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:33.800786 6924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:33.800786 6921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:33.800951 6916 server_base.cc:1034] running on GCE node
W20260619 14:55:33.800845 6922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:33.801432 6916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:33.801954 6916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:33.803134 6916 hybrid_clock.cc:648] HybridClock initialized: now 1781880933803112 us; error 33 us; skew 500 ppm
I20260619 14:55:33.804811 6916 webserver.cc:460] Webserver started at http://127.19.34.65:37757/ using document root <none> and password file <none>
I20260619 14:55:33.805311 6916 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:33.805399 6916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:33.805594 6916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:33.807284 6916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-0/data/instance:
uuid: "7af8ec95ad3545b0af042060715fa0ca"
format_stamp: "Formatted at 2026-06-19 14:55:33 on dist-test-slave-3qdh"
server_key: "a81c6589acc275d143826014a3abe76e"
server_key_iv: "77ae9d7ab5a16364186629b7314459f6"
server_key_version: "encryptionkey@0"
I20260619 14:55:33.807776 6916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance:
uuid: "7af8ec95ad3545b0af042060715fa0ca"
format_stamp: "Formatted at 2026-06-19 14:55:33 on dist-test-slave-3qdh"
server_key: "a81c6589acc275d143826014a3abe76e"
server_key_iv: "77ae9d7ab5a16364186629b7314459f6"
server_key_version: "encryptionkey@0"
I20260619 14:55:33.811175 6916 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260619 14:55:33.813324 6930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:33.814348 6916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:55:33.814465 6916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-0/wal
uuid: "7af8ec95ad3545b0af042060715fa0ca"
format_stamp: "Formatted at 2026-06-19 14:55:33 on dist-test-slave-3qdh"
server_key: "a81c6589acc275d143826014a3abe76e"
server_key_iv: "77ae9d7ab5a16364186629b7314459f6"
server_key_version: "encryptionkey@0"
I20260619 14:55:33.814559 6916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:33.831583 6916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:33.832069 6916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:33.832242 6916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:33.832772 6916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:55:33.833564 6916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:55:33.833639 6916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:33.833709 6916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:55:33.833755 6916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:33.841485 6916 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.65:34811
I20260619 14:55:33.841573 7043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.65:34811 every 8 connection(s)
I20260619 14:55:33.842182 6916 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-0/data/info.pb
I20260619 14:55:33.848138 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 6916
I20260619 14:55:33.848286 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-0/wal/instance
I20260619 14:55:33.848604 19593 external_mini_cluster.cc:1428] Setting key 82364fa386e85ffb69a84a3e8981cd44
I20260619 14:55:33.850492 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.66:0
--local_ip_for_outbound_sockets=127.19.34.66
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:40949
--builtin_ntp_servers=127.19.34.84:39629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:55:33.850809 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:33.843874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.65:42345 (local address 127.19.34.126:40949)
0619 14:55:33.844053 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:33.844056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:33.844969 (+ 913us) server_negotiation.cc:407] Connection header received
0619 14:55:33.846249 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:33.846252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:33.846304 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:33.846398 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:33.847153 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.847625 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.848960 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.849196 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.849305 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:33.850313 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:33.850317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:33.850319 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:33.850332 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:33.850354 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:33.850362 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:33.850449 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:33.850594 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 14:55:33.850670 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 14:55:33.851444 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:33.844218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:40949 (local address 127.19.34.65:42345)
0619 14:55:33.844841 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:33.844874 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:33.846053 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:33.846539 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:33.846562 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:33.846633 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:33.847018 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:33.847029 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.847790 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.847793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:33.848833 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:33.848843 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:33.849336 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:33.849340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:33.849485 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:33.850068 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:55:33.850070 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:33.850101 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:55:33.850159 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:33.850450 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:33.850461 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:33.850466 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:55:33.850584 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 14:55:33.850809 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":118,"threads_started":1}
I20260619 14:55:33.852581 7044 heartbeater.cc:347] Connected to a master server at 127.19.34.126:40949
I20260619 14:55:33.852816 7044 heartbeater.cc:464] Registering TS with master...
I20260619 14:55:33.853287 7044 heartbeater.cc:511] Master 127.19.34.126:40949 requested a full tablet report, sending...
I20260619 14:55:33.854780 6856 ts_manager.cc:194] Registered new tserver with Master: 7af8ec95ad3545b0af042060715fa0ca (127.19.34.65:34811)
I20260619 14:55:33.856452 6856 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.34.65:42345
W20260619 14:55:33.951223 7047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:33.951424 7047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:33.951458 7047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:33.954965 7047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:33.955008 7047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:33.955053 7047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.66
I20260619 14:55:33.959295 7047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:39629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.34.66
--webserver_port=0
--tserver_master_addrs=127.19.34.126:40949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.66
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:33.960021 7047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:33.961108 7047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:33.969861 7053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:33.969833 7055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:33.969837 7052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:33.969944 7047 server_base.cc:1034] running on GCE node
I20260619 14:55:33.970369 7047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:33.970925 7047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:33.972086 7047 hybrid_clock.cc:648] HybridClock initialized: now 1781880933972069 us; error 29 us; skew 500 ppm
I20260619 14:55:33.973853 7047 webserver.cc:460] Webserver started at http://127.19.34.66:37437/ using document root <none> and password file <none>
I20260619 14:55:33.974399 7047 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:33.974453 7047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:33.974677 7047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:33.976536 7047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-1/data/instance:
uuid: "70cb4113e4da48a0be7b11240d8cc4e9"
format_stamp: "Formatted at 2026-06-19 14:55:33 on dist-test-slave-3qdh"
server_key: "19c2522468cfed182e00554dd79bb148"
server_key_iv: "648f6e4aa510a3da8664bf6898d78abc"
server_key_version: "encryptionkey@0"
I20260619 14:55:33.977035 7047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance:
uuid: "70cb4113e4da48a0be7b11240d8cc4e9"
format_stamp: "Formatted at 2026-06-19 14:55:33 on dist-test-slave-3qdh"
server_key: "19c2522468cfed182e00554dd79bb148"
server_key_iv: "648f6e4aa510a3da8664bf6898d78abc"
server_key_version: "encryptionkey@0"
I20260619 14:55:33.980234 7047 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:55:33.982385 7061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:33.983448 7047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:33.983573 7047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-1/wal
uuid: "70cb4113e4da48a0be7b11240d8cc4e9"
format_stamp: "Formatted at 2026-06-19 14:55:33 on dist-test-slave-3qdh"
server_key: "19c2522468cfed182e00554dd79bb148"
server_key_iv: "648f6e4aa510a3da8664bf6898d78abc"
server_key_version: "encryptionkey@0"
I20260619 14:55:33.983675 7047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:33.998898 7047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:33.999400 7047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:33.999563 7047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:34.000015 7047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:55:34.000831 7047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:55:34.000871 7047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:34.000901 7047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:55:34.000913 7047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:34.009196 7047 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.66:37203
I20260619 14:55:34.009272 7174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.66:37203 every 8 connection(s)
I20260619 14:55:34.009936 7047 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-1/data/info.pb
I20260619 14:55:34.017266 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 7047
I20260619 14:55:34.017386 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-1/wal/instance
I20260619 14:55:34.017724 19593 external_mini_cluster.cc:1428] Setting key 33e8780e42e5c732042a7f67fdb19b62
I20260619 14:55:34.017727 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:34.011554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:40711 (local address 127.19.34.126:40949)
0619 14:55:34.011690 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:34.011694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:34.012595 (+ 901us) server_negotiation.cc:407] Connection header received
0619 14:55:34.013867 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:34.013870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:34.013927 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:34.014027 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:34.014745 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.015186 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.016105 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.016283 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.016414 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:34.017244 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:34.017247 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:34.017250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:34.017263 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:34.017284 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:34.017291 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:34.017366 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:34.017543 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 14:55:34.017599 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:55:34.018497 7177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:34.011876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:40949 (local address 127.19.34.66:40711)
0619 14:55:34.012487 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:34.012521 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:34.013685 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:34.014162 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:34.014168 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:34.014234 (+ 66us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:34.014605 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:34.014615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.015290 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.015292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:34.016002 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:34.016010 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.016456 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.016459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:34.016571 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:34.017025 (+ 454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:55:34.017027 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:34.017057 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:55:34.017105 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:34.017378 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:34.017387 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:34.017392 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:55:34.017514 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 14:55:34.017745 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":102,"threads_started":1}
I20260619 14:55:34.019522 7175 heartbeater.cc:347] Connected to a master server at 127.19.34.126:40949
I20260619 14:55:34.019734 7175 heartbeater.cc:464] Registering TS with master...
I20260619 14:55:34.019893 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.34.67:0
--local_ip_for_outbound_sockets=127.19.34.67
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:40949
--builtin_ntp_servers=127.19.34.84:39629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:55:34.020236 7175 heartbeater.cc:511] Master 127.19.34.126:40949 requested a full tablet report, sending...
I20260619 14:55:34.021152 6856 ts_manager.cc:194] Registered new tserver with Master: 70cb4113e4da48a0be7b11240d8cc4e9 (127.19.34.66:37203)
I20260619 14:55:34.021628 6856 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.34.66:40711
W20260619 14:55:34.115527 7178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:34.115710 7178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:34.115741 7178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:34.119367 7178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:34.119406 7178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:34.119467 7178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.34.67
I20260619 14:55:34.123521 7178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:39629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.34.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.34.67
--webserver_port=0
--tserver_master_addrs=127.19.34.126:40949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.34.67
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:34.124331 7178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:34.125375 7178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:34.133632 7186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:34.133627 7184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:34.133698 7178 server_base.cc:1034] running on GCE node
W20260619 14:55:34.133627 7183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:34.134212 7178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:34.134684 7178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:34.135861 7178 hybrid_clock.cc:648] HybridClock initialized: now 1781880934135851 us; error 27 us; skew 500 ppm
I20260619 14:55:34.137346 7178 webserver.cc:460] Webserver started at http://127.19.34.67:43731/ using document root <none> and password file <none>
I20260619 14:55:34.137800 7178 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:34.137884 7178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:34.138103 7178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:34.139771 7178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-2/data/instance:
uuid: "d35dd400543449ec8bd07072cffdcfdd"
format_stamp: "Formatted at 2026-06-19 14:55:34 on dist-test-slave-3qdh"
server_key: "7cd0b9872009e5a4eadeb12144fb48e9"
server_key_iv: "11d4d283770a49ce7915388ddf2a1d38"
server_key_version: "encryptionkey@0"
I20260619 14:55:34.140244 7178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance:
uuid: "d35dd400543449ec8bd07072cffdcfdd"
format_stamp: "Formatted at 2026-06-19 14:55:34 on dist-test-slave-3qdh"
server_key: "7cd0b9872009e5a4eadeb12144fb48e9"
server_key_iv: "11d4d283770a49ce7915388ddf2a1d38"
server_key_version: "encryptionkey@0"
I20260619 14:55:34.143361 7178 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:55:34.145543 7192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:34.146663 7178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:55:34.146780 7178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-2/wal
uuid: "d35dd400543449ec8bd07072cffdcfdd"
format_stamp: "Formatted at 2026-06-19 14:55:34 on dist-test-slave-3qdh"
server_key: "7cd0b9872009e5a4eadeb12144fb48e9"
server_key_iv: "11d4d283770a49ce7915388ddf2a1d38"
server_key_version: "encryptionkey@0"
I20260619 14:55:34.146873 7178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:34.158885 7178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:34.159374 7178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:34.159540 7178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:34.160091 7178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:55:34.160941 7178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:55:34.161021 7178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:34.161090 7178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:55:34.161124 7178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:34.169448 7178 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.67:41557
I20260619 14:55:34.169477 7305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.67:41557 every 8 connection(s)
I20260619 14:55:34.170146 7178 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-2/data/info.pb
I20260619 14:55:34.175798 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 7178
I20260619 14:55:34.175907 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-2/wal/instance
I20260619 14:55:34.176143 19593 external_mini_cluster.cc:1428] Setting key 56fa93ad0a23cf8ec0f49b0b6ed162c3
I20260619 14:55:34.178458 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:34.171629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.67:45353 (local address 127.19.34.126:40949)
0619 14:55:34.171794 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:34.171799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:34.172715 (+ 916us) server_negotiation.cc:407] Connection header received
0619 14:55:34.173909 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:34.173912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:34.173955 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:34.174038 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:34.174751 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.175240 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.176644 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.176956 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.177073 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:34.178024 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:34.178027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:34.178030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:34.178043 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:34.178064 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:34.178074 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:34.178162 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:34.178317 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 14:55:34.178362 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:55:34.179138 7308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:34.171983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:40949 (local address 127.19.34.67:45353)
0619 14:55:34.172582 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:34.172615 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:34.173740 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:34.174177 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:34.174182 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:34.174250 (+ 68us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:34.174627 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:34.174637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.175347 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.175350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:34.176519 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:34.176527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.177098 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.177102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:34.177257 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:34.177757 (+ 500us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:55:34.177759 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:34.177823 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:55:34.177889 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:34.178197 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:34.178206 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:34.178212 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:55:34.178305 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 14:55:34.178478 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":111,"threads_started":1}
I20260619 14:55:34.180142 7306 heartbeater.cc:347] Connected to a master server at 127.19.34.126:40949
I20260619 14:55:34.180373 7306 heartbeater.cc:464] Registering TS with master...
I20260619 14:55:34.180811 7306 heartbeater.cc:511] Master 127.19.34.126:40949 requested a full tablet report, sending...
I20260619 14:55:34.181711 6856 ts_manager.cc:194] Registered new tserver with Master: d35dd400543449ec8bd07072cffdcfdd (127.19.34.67:41557)
I20260619 14:55:34.182240 6856 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.34.67:45353
I20260619 14:55:34.189745 19593 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:55:34.194905 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:34.191590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48576 (local address 127.19.34.126:40949)
0619 14:55:34.191722 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:34.191727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:34.191904 (+ 177us) server_negotiation.cc:407] Connection header received
0619 14:55:34.191962 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:34.191964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:34.191997 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:34.192058 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:34.192335 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.192817 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.193611 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.193803 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.193875 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:34.194535 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:34.194537 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:34.194539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:34.194548 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:34.194563 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:34.194569 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:34.194628 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:34.194712 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 14:55:34.194748 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:55:34.197394 6856 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:55:34.200011 6856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:55:34.212059 7318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:34.206933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34811 (local address 127.0.0.1:38054)
0619 14:55:34.207307 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:34.207344 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:34.207440 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:34.208283 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:34.208288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:34.208329 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:34.208431 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:34.208440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.209582 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.209585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:34.210383 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:34.210391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.211538 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.211542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:34.211632 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:34.211656 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:55:34.211769 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 14:55:34.211922 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":55,"threads_started":1}
I20260619 14:55:34.212069 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:34.207047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38054 (local address 127.19.34.65:34811)
0619 14:55:34.207402 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:34.207443 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:34.207466 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:55:34.207558 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:34.207564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:34.207740 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:34.208108 (+ 368us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:34.208561 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.209472 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.210498 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.211228 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.211315 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:34.211477 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:34.211812 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 14:55:34.211921 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":108,"threads_started":1}
I20260619 14:55:34.212854 7317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:34.207034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.66:37203 (local address 127.0.0.1:43366)
0619 14:55:34.207840 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:34.207853 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:34.207953 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:34.208867 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:34.208871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:34.208892 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:34.209006 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:34.209013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.210295 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.210298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:34.211307 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:34.211315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.212421 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.212424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:34.212569 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:34.212594 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:55:34.212656 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 14:55:34.212721 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"mutex_wait_us":41,"thread_start_us":109,"threads_started":1}
I20260619 14:55:34.213076 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:34.207788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43366 (local address 127.19.34.66:37203)
0619 14:55:34.208416 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:34.208428 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:34.208451 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:55:34.208526 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:34.208532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:34.208722 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:34.208866 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:34.209158 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.210180 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.211457 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.212254 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.212581 (+ 327us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:34.212753 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:34.212793 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 14:55:34.212875 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":152,"threads_started":1}
I20260619 14:55:34.214954 6978 tablet_service.cc:1467] Processing CreateTablet for tablet 4c4b4090d9b24de3b5399e20871c8993 (DEFAULT_TABLE table=test-table [id=0c16cc2a340a441c85c039c73a7a6795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:55:34.215305 7109 tablet_service.cc:1467] Processing CreateTablet for tablet 4c4b4090d9b24de3b5399e20871c8993 (DEFAULT_TABLE table=test-table [id=0c16cc2a340a441c85c039c73a7a6795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:55:34.215745 6978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c4b4090d9b24de3b5399e20871c8993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:34.216073 7109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c4b4090d9b24de3b5399e20871c8993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:34.216550 7320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:34.207456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:41557 (local address 127.0.0.1:38560)
0619 14:55:34.208155 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:34.208166 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:34.208257 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:34.208904 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:34.208907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:34.208919 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:34.209040 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:34.209050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.210317 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.210320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:34.211450 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:34.211459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.212413 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.212417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:34.213074 (+ 657us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:34.213112 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 14:55:34.216385 (+ 3273us) client_negotiation.cc:240] Negotiation successful
0619 14:55:34.216448 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":127,"thread_start_us":79,"threads_started":1}
I20260619 14:55:34.216902 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:34.207581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38560 (local address 127.19.34.67:41557)
0619 14:55:34.208478 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:34.208499 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:34.208520 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:55:34.208579 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:34.208585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:34.208751 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:34.209020 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:34.209166 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.210180 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.211561 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.212254 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.212410 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:34.212573 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:34.216703 (+ 4130us) server_negotiation.cc:299] Negotiation successful
0619 14:55:34.216779 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":808,"thread_start_us":107,"threads_started":1}
I20260619 14:55:34.219079 7240 tablet_service.cc:1467] Processing CreateTablet for tablet 4c4b4090d9b24de3b5399e20871c8993 (DEFAULT_TABLE table=test-table [id=0c16cc2a340a441c85c039c73a7a6795]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:55:34.219902 7240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c4b4090d9b24de3b5399e20871c8993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:34.221685 7323 tablet_bootstrap.cc:492] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9: Bootstrap starting.
I20260619 14:55:34.222440 7324 tablet_bootstrap.cc:492] T 4c4b4090d9b24de3b5399e20871c8993 P 7af8ec95ad3545b0af042060715fa0ca: Bootstrap starting.
I20260619 14:55:34.224030 7323 tablet_bootstrap.cc:654] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:34.224414 7324 tablet_bootstrap.cc:654] T 4c4b4090d9b24de3b5399e20871c8993 P 7af8ec95ad3545b0af042060715fa0ca: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:34.224802 7323 log.cc:824] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:34.225106 7324 log.cc:824] T 4c4b4090d9b24de3b5399e20871c8993 P 7af8ec95ad3545b0af042060715fa0ca: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:34.226450 7323 tablet_bootstrap.cc:492] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9: No bootstrap required, opened a new log
I20260619 14:55:34.226627 7323 ts_tablet_manager.cc:1397] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:55:34.226694 7324 tablet_bootstrap.cc:492] T 4c4b4090d9b24de3b5399e20871c8993 P 7af8ec95ad3545b0af042060715fa0ca: No bootstrap required, opened a new log
I20260619 14:55:34.226854 7324 ts_tablet_manager.cc:1397] T 4c4b4090d9b24de3b5399e20871c8993 P 7af8ec95ad3545b0af042060715fa0ca: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 14:55:34.228410 7327 tablet_bootstrap.cc:492] T 4c4b4090d9b24de3b5399e20871c8993 P d35dd400543449ec8bd07072cffdcfdd: Bootstrap starting.
I20260619 14:55:34.230041 7323 raft_consensus.cc:348] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7af8ec95ad3545b0af042060715fa0ca" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34811 } } peers { permanent_uuid: "70cb4113e4da48a0be7b11240d8cc4e9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 37203 } } peers { permanent_uuid: "d35dd400543449ec8bd07072cffdcfdd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41557 } }
I20260619 14:55:34.230239 7323 raft_consensus.cc:374] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:34.230243 7327 tablet_bootstrap.cc:654] T 4c4b4090d9b24de3b5399e20871c8993 P d35dd400543449ec8bd07072cffdcfdd: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:34.230285 7323 raft_consensus.cc:729] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70cb4113e4da48a0be7b11240d8cc4e9, State: Initialized, Role: FOLLOWER
I20260619 14:55:34.230650 7323 consensus_queue.cc:260] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7af8ec95ad3545b0af042060715fa0ca" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34811 } } peers { permanent_uuid: "70cb4113e4da48a0be7b11240d8cc4e9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 37203 } } peers { permanent_uuid: "d35dd400543449ec8bd07072cffdcfdd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41557 } }
I20260619 14:55:34.230619 7324 raft_consensus.cc:348] T 4c4b4090d9b24de3b5399e20871c8993 P 7af8ec95ad3545b0af042060715fa0ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7af8ec95ad3545b0af042060715fa0ca" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34811 } } peers { permanent_uuid: "70cb4113e4da48a0be7b11240d8cc4e9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 37203 } } peers { permanent_uuid: "d35dd400543449ec8bd07072cffdcfdd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41557 } }
I20260619 14:55:34.230836 7324 raft_consensus.cc:374] T 4c4b4090d9b24de3b5399e20871c8993 P 7af8ec95ad3545b0af042060715fa0ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:34.230883 7324 raft_consensus.cc:729] T 4c4b4090d9b24de3b5399e20871c8993 P 7af8ec95ad3545b0af042060715fa0ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af8ec95ad3545b0af042060715fa0ca, State: Initialized, Role: FOLLOWER
I20260619 14:55:34.230911 7327 log.cc:824] T 4c4b4090d9b24de3b5399e20871c8993 P d35dd400543449ec8bd07072cffdcfdd: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:34.231246 7175 heartbeater.cc:503] Master 127.19.34.126:40949 was elected leader, sending a full tablet report...
I20260619 14:55:34.231302 7324 consensus_queue.cc:260] T 4c4b4090d9b24de3b5399e20871c8993 P 7af8ec95ad3545b0af042060715fa0ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7af8ec95ad3545b0af042060715fa0ca" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34811 } } peers { permanent_uuid: "70cb4113e4da48a0be7b11240d8cc4e9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 37203 } } peers { permanent_uuid: "d35dd400543449ec8bd07072cffdcfdd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41557 } }
I20260619 14:55:34.231518 7323 ts_tablet_manager.cc:1428] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:55:34.231990 7324 ts_tablet_manager.cc:1428] T 4c4b4090d9b24de3b5399e20871c8993 P 7af8ec95ad3545b0af042060715fa0ca: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260619 14:55:34.232283 7044 heartbeater.cc:503] Master 127.19.34.126:40949 was elected leader, sending a full tablet report...
I20260619 14:55:34.233635 7327 tablet_bootstrap.cc:492] T 4c4b4090d9b24de3b5399e20871c8993 P d35dd400543449ec8bd07072cffdcfdd: No bootstrap required, opened a new log
I20260619 14:55:34.233807 7327 ts_tablet_manager.cc:1397] T 4c4b4090d9b24de3b5399e20871c8993 P d35dd400543449ec8bd07072cffdcfdd: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260619 14:55:34.236831 7327 raft_consensus.cc:348] T 4c4b4090d9b24de3b5399e20871c8993 P d35dd400543449ec8bd07072cffdcfdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7af8ec95ad3545b0af042060715fa0ca" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34811 } } peers { permanent_uuid: "70cb4113e4da48a0be7b11240d8cc4e9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 37203 } } peers { permanent_uuid: "d35dd400543449ec8bd07072cffdcfdd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41557 } }
I20260619 14:55:34.236981 7327 raft_consensus.cc:374] T 4c4b4090d9b24de3b5399e20871c8993 P d35dd400543449ec8bd07072cffdcfdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:34.237058 7327 raft_consensus.cc:729] T 4c4b4090d9b24de3b5399e20871c8993 P d35dd400543449ec8bd07072cffdcfdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d35dd400543449ec8bd07072cffdcfdd, State: Initialized, Role: FOLLOWER
I20260619 14:55:34.237341 7327 consensus_queue.cc:260] T 4c4b4090d9b24de3b5399e20871c8993 P d35dd400543449ec8bd07072cffdcfdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7af8ec95ad3545b0af042060715fa0ca" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34811 } } peers { permanent_uuid: "70cb4113e4da48a0be7b11240d8cc4e9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 37203 } } peers { permanent_uuid: "d35dd400543449ec8bd07072cffdcfdd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41557 } }
I20260619 14:55:34.237814 7306 heartbeater.cc:503] Master 127.19.34.126:40949 was elected leader, sending a full tablet report...
I20260619 14:55:34.238027 7327 ts_tablet_manager.cc:1428] T 4c4b4090d9b24de3b5399e20871c8993 P d35dd400543449ec8bd07072cffdcfdd: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260619 14:55:34.260705 7176 tablet.cc:2369] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:55:34.305166 7328 raft_consensus.cc:482] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:55:34.305371 7328 raft_consensus.cc:504] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7af8ec95ad3545b0af042060715fa0ca" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34811 } } peers { permanent_uuid: "70cb4113e4da48a0be7b11240d8cc4e9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 37203 } } peers { permanent_uuid: "d35dd400543449ec8bd07072cffdcfdd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41557 } }
I20260619 14:55:34.306335 7328 leader_election.cc:288] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7af8ec95ad3545b0af042060715fa0ca (127.19.34.65:34811), d35dd400543449ec8bd07072cffdcfdd (127.19.34.67:41557)
I20260619 14:55:34.310544 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:34.306628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:38207 (local address 127.19.34.67:41557)
0619 14:55:34.306752 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:34.306756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:34.306771 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:55:34.306844 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:34.306850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:34.306898 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:34.306972 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:34.307287 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.308037 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.309464 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.309980 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.310088 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:34.310130 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:34.310361 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 14:55:34.310412 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:55:34.310565 7177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:34.306528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.67:41557 (local address 127.19.34.66:38207)
0619 14:55:34.306660 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:34.306675 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:34.306759 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:34.307005 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:34.307008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:34.307028 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:34.307178 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:34.307188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.308153 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.308155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:34.309355 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:34.309364 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.310117 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.310119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:34.310192 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:34.310251 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 14:55:34.310326 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 14:55:34.310426 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":23}
I20260619 14:55:34.310691 7332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:34.306528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34811 (local address 127.19.34.66:54417)
0619 14:55:34.307052 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:34.307064 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:34.307151 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:34.307399 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:34.307400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:34.307413 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:55:34.307497 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:34.307502 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.308429 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.308432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:34.309518 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:34.309524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.310271 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.310277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:34.310388 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:34.310405 (+ 17us) client_negotiation.cc:769] Sending connection context
0619 14:55:34.310466 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 14:55:34.310514 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":197,"thread_start_us":112,"threads_started":1}
I20260619 14:55:34.310734 7319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:34.306681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.34.66:54417 (local address 127.19.34.65:34811)
0619 14:55:34.306792 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:34.306798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:34.307145 (+ 347us) server_negotiation.cc:407] Connection header received
0619 14:55:34.307261 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:34.307263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:34.307310 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:34.307376 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:55:34.307601 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.308275 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.309621 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.310139 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.310239 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:34.310267 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:34.310502 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 14:55:34.310605 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:55:34.311223 7260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4b4090d9b24de3b5399e20871c8993" candidate_uuid: "70cb4113e4da48a0be7b11240d8cc4e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35dd400543449ec8bd07072cffdcfdd" is_pre_election: true
I20260619 14:55:34.311235 6998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4b4090d9b24de3b5399e20871c8993" candidate_uuid: "70cb4113e4da48a0be7b11240d8cc4e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8ec95ad3545b0af042060715fa0ca" is_pre_election: true
I20260619 14:55:34.311460 7260 raft_consensus.cc:2454] T 4c4b4090d9b24de3b5399e20871c8993 P d35dd400543449ec8bd07072cffdcfdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70cb4113e4da48a0be7b11240d8cc4e9 in term 0.
I20260619 14:55:34.311479 6998 raft_consensus.cc:2454] T 4c4b4090d9b24de3b5399e20871c8993 P 7af8ec95ad3545b0af042060715fa0ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70cb4113e4da48a0be7b11240d8cc4e9 in term 0.
I20260619 14:55:34.311911 7062 leader_election.cc:302] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70cb4113e4da48a0be7b11240d8cc4e9, d35dd400543449ec8bd07072cffdcfdd; no voters:
I20260619 14:55:34.312166 7328 raft_consensus.cc:2781] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:55:34.312237 7328 raft_consensus.cc:482] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:55:34.312274 7328 raft_consensus.cc:3037] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:34.313230 7328 raft_consensus.cc:504] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7af8ec95ad3545b0af042060715fa0ca" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34811 } } peers { permanent_uuid: "70cb4113e4da48a0be7b11240d8cc4e9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 37203 } } peers { permanent_uuid: "d35dd400543449ec8bd07072cffdcfdd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41557 } }
I20260619 14:55:34.313511 7328 leader_election.cc:288] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 [CANDIDATE]: Term 1 election: Requested vote from peers 7af8ec95ad3545b0af042060715fa0ca (127.19.34.65:34811), d35dd400543449ec8bd07072cffdcfdd (127.19.34.67:41557)
I20260619 14:55:34.313807 6998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4b4090d9b24de3b5399e20871c8993" candidate_uuid: "70cb4113e4da48a0be7b11240d8cc4e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7af8ec95ad3545b0af042060715fa0ca"
I20260619 14:55:34.313848 7260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4b4090d9b24de3b5399e20871c8993" candidate_uuid: "70cb4113e4da48a0be7b11240d8cc4e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d35dd400543449ec8bd07072cffdcfdd"
I20260619 14:55:34.313921 6998 raft_consensus.cc:3037] T 4c4b4090d9b24de3b5399e20871c8993 P 7af8ec95ad3545b0af042060715fa0ca [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:34.313937 7260 raft_consensus.cc:3037] T 4c4b4090d9b24de3b5399e20871c8993 P d35dd400543449ec8bd07072cffdcfdd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:34.315121 7260 raft_consensus.cc:2454] T 4c4b4090d9b24de3b5399e20871c8993 P d35dd400543449ec8bd07072cffdcfdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70cb4113e4da48a0be7b11240d8cc4e9 in term 1.
I20260619 14:55:34.315130 6998 raft_consensus.cc:2454] T 4c4b4090d9b24de3b5399e20871c8993 P 7af8ec95ad3545b0af042060715fa0ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70cb4113e4da48a0be7b11240d8cc4e9 in term 1.
I20260619 14:55:34.315449 7062 leader_election.cc:302] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70cb4113e4da48a0be7b11240d8cc4e9, d35dd400543449ec8bd07072cffdcfdd; no voters:
I20260619 14:55:34.315653 7328 raft_consensus.cc:2781] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:34.315894 7328 raft_consensus.cc:686] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 [term 1 LEADER]: Becoming Leader. State: Replica: 70cb4113e4da48a0be7b11240d8cc4e9, State: Running, Role: LEADER
I20260619 14:55:34.316185 7328 consensus_queue.cc:237] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7af8ec95ad3545b0af042060715fa0ca" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34811 } } peers { permanent_uuid: "70cb4113e4da48a0be7b11240d8cc4e9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 37203 } } peers { permanent_uuid: "d35dd400543449ec8bd07072cffdcfdd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41557 } }
I20260619 14:55:34.318859 6855 catalog_manager.cc:5495] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70cb4113e4da48a0be7b11240d8cc4e9 (127.19.34.66). New cstate: current_term: 1 leader_uuid: "70cb4113e4da48a0be7b11240d8cc4e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7af8ec95ad3545b0af042060715fa0ca" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70cb4113e4da48a0be7b11240d8cc4e9" member_type: VOTER last_known_addr { host: "127.19.34.66" port: 37203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d35dd400543449ec8bd07072cffdcfdd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41557 } health_report { overall_health: UNKNOWN } } }
I20260619 14:55:34.330716 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:34.327274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.19.34.66:37203)
0619 14:55:34.327398 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:34.327401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:34.327412 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:55:34.327543 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:34.327545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:34.327604 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:34.327707 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:55:34.327960 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.328507 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.329370 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:34.329605 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:34.329720 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:34.329997 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:55:34.330338 (+ 341us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:55:34.330460 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:34.330516 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 14:55:34.330562 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:55:34.335850 7260 raft_consensus.cc:1261] T 4c4b4090d9b24de3b5399e20871c8993 P d35dd400543449ec8bd07072cffdcfdd [term 1 FOLLOWER]: Refusing update from remote peer 70cb4113e4da48a0be7b11240d8cc4e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:55:34.336431 6998 raft_consensus.cc:1261] T 4c4b4090d9b24de3b5399e20871c8993 P 7af8ec95ad3545b0af042060715fa0ca [term 1 FOLLOWER]: Refusing update from remote peer 70cb4113e4da48a0be7b11240d8cc4e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:55:34.336544 7328 consensus_queue.cc:1035] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d35dd400543449ec8bd07072cffdcfdd" member_type: VOTER last_known_addr { host: "127.19.34.67" port: 41557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:55:34.336977 7333 consensus_queue.cc:1035] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7af8ec95ad3545b0af042060715fa0ca" member_type: VOTER last_known_addr { host: "127.19.34.65" port: 34811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:55:34.343699 7337 mvcc.cc:205] Tried to move back new op lower bound from 7298584307034071040 to 7298584306961330176. Current Snapshot: MvccSnapshot[applied={T|T < 7298584307034071040}]
I20260619 14:55:34.352248 6855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:55:34.352409 6855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:55:34.354411 6855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 97a9680b4cc64a43a645e2c234b4d40d: Sending DeleteTablet for 3 replicas of tablet 4c4b4090d9b24de3b5399e20871c8993
I20260619 14:55:34.355023 6978 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c4b4090d9b24de3b5399e20871c8993 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:55:34 UTC) from {username='slave'} at 127.0.0.1:38054
I20260619 14:55:34.355101 7109 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c4b4090d9b24de3b5399e20871c8993 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:55:34 UTC) from {username='slave'} at 127.0.0.1:43366
I20260619 14:55:34.355348 7240 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c4b4090d9b24de3b5399e20871c8993 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:55:34 UTC) from {username='slave'} at 127.0.0.1:38560
I20260619 14:55:34.355567 7346 tablet_replica.cc:330] T 4c4b4090d9b24de3b5399e20871c8993 P 7af8ec95ad3545b0af042060715fa0ca: stopping tablet replica
I20260619 14:55:34.355693 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 6916
I20260619 14:55:34.355815 7346 raft_consensus.cc:2229] T 4c4b4090d9b24de3b5399e20871c8993 P 7af8ec95ad3545b0af042060715fa0ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:55:34.356129 7347 tablet_replica.cc:330] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9: stopping tablet replica
I20260619 14:55:34.356369 7347 raft_consensus.cc:2229] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:55:34.356670 7347 raft_consensus.cc:2258] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:55:34.357797 7347 ts_tablet_manager.cc:1905] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:55:34.360394 7347 ts_tablet_manager.cc:1918] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:55:34.360482 7347 log.cc:1196] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-1/wal/wals/4c4b4090d9b24de3b5399e20871c8993
I20260619 14:55:34.360778 7347 ts_tablet_manager.cc:1939] T 4c4b4090d9b24de3b5399e20871c8993 P 70cb4113e4da48a0be7b11240d8cc4e9: Deleting consensus metadata
I20260619 14:55:34.361707 7348 tablet_replica.cc:330] T 4c4b4090d9b24de3b5399e20871c8993 P d35dd400543449ec8bd07072cffdcfdd: stopping tablet replica
I20260619 14:55:34.361799 6840 catalog_manager.cc:4841] TS 70cb4113e4da48a0be7b11240d8cc4e9 (127.19.34.66:37203): tablet 4c4b4090d9b24de3b5399e20871c8993 (table test-table [id=0c16cc2a340a441c85c039c73a7a6795]) successfully deleted
I20260619 14:55:34.361968 7348 raft_consensus.cc:2229] T 4c4b4090d9b24de3b5399e20871c8993 P d35dd400543449ec8bd07072cffdcfdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:55:34.362217 7348 raft_consensus.cc:2258] T 4c4b4090d9b24de3b5399e20871c8993 P d35dd400543449ec8bd07072cffdcfdd [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 14:55:34.363313 6842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.34.65:34811 (error 108)
I20260619 14:55:34.363595 7348 ts_tablet_manager.cc:1905] T 4c4b4090d9b24de3b5399e20871c8993 P d35dd400543449ec8bd07072cffdcfdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:55:34.364110 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 7047
I20260619 14:55:34.364245 7320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:34.363935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.65:34811 (local address 127.0.0.1:38066)
0619 14:55:34.364080 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:34.364161 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.65:34811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260619 14:55:34.364645 6842 catalog_manager.cc:4584] TS 7af8ec95ad3545b0af042060715fa0ca (127.19.34.65:34811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c4b4090d9b24de3b5399e20871c8993: Network error: Client connection negotiation failed: client connection to 127.19.34.65:34811: connect: Connection refused (error 111)
I20260619 14:55:34.367138 7348 ts_tablet_manager.cc:1918] T 4c4b4090d9b24de3b5399e20871c8993 P d35dd400543449ec8bd07072cffdcfdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:55:34.367252 7348 log.cc:1196] T 4c4b4090d9b24de3b5399e20871c8993 P d35dd400543449ec8bd07072cffdcfdd: Deleting WAL directory at /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880697016847-19593-0/minicluster-data/ts-2/wal/wals/4c4b4090d9b24de3b5399e20871c8993
I20260619 14:55:34.367501 7348 ts_tablet_manager.cc:1939] T 4c4b4090d9b24de3b5399e20871c8993 P d35dd400543449ec8bd07072cffdcfdd: Deleting consensus metadata
I20260619 14:55:34.368464 6840 catalog_manager.cc:4841] TS d35dd400543449ec8bd07072cffdcfdd (127.19.34.67:41557): tablet 4c4b4090d9b24de3b5399e20871c8993 (table test-table [id=0c16cc2a340a441c85c039c73a7a6795]) successfully deleted
I20260619 14:55:34.369485 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 7178
I20260619 14:55:34.374480 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 6825
2026-06-19T14:55:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (928 ms)
[----------] 6 tests from EncryptionPolicyTest (9480 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:34 dist-test-slave-3qdh krb5kdc[7352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:34 dist-test-slave-3qdh krb5kdc[7352](info): set up 2 sockets
Jun 19 14:55:34 dist-test-slave-3qdh krb5kdc[7352](info): commencing operation
krb5kdc: starting...
W20260619 14:55:35.324163 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:35 dist-test-slave-3qdh krb5kdc[7352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:55:35.432186 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41531
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:35.526777 7365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:35.526996 7365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:35.527060 7365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:35.530753 7365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:35.530833 7365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:35.530866 7365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:35.530923 7365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:35.530957 7365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:35.535454 7365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:35.536383 7365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:35.537179 7365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:35.544224 7370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:35.544229 7371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:35.544288 7373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:35.544368 7365 server_base.cc:1034] running on GCE node
I20260619 14:55:35.544863 7365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:35.545701 7365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:35.546949 7365 hybrid_clock.cc:648] HybridClock initialized: now 1781880935546916 us; error 30 us; skew 500 ppm
Jun 19 14:55:35 dist-test-slave-3qdh krb5kdc[7352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:35.549451 7365 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:55:35.550184 7365 webserver.cc:460] Webserver started at http://127.0.0.1:40483/ using document root <none> and password file <none>
I20260619 14:55:35.550567 7365 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:35.550616 7365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:35.550760 7365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:35.552430 7365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "3dc5e2bec3ae4d05ad86896dad27db0e"
format_stamp: "Formatted at 2026-06-19 14:55:35 on dist-test-slave-3qdh"
server_key: "e3e86c3f0916178bce45d6b028d5d40c"
server_key_iv: "0a564fbe6eb853e750a1ed0b410dec62"
server_key_version: "encryptionkey@0"
I20260619 14:55:35.552839 7365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "3dc5e2bec3ae4d05ad86896dad27db0e"
format_stamp: "Formatted at 2026-06-19 14:55:35 on dist-test-slave-3qdh"
server_key: "e3e86c3f0916178bce45d6b028d5d40c"
server_key_iv: "0a564fbe6eb853e750a1ed0b410dec62"
server_key_version: "encryptionkey@0"
I20260619 14:55:35.555745 7365 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:55:35.557739 7380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:35.558734 7365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:55:35.558861 7365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "3dc5e2bec3ae4d05ad86896dad27db0e"
format_stamp: "Formatted at 2026-06-19 14:55:35 on dist-test-slave-3qdh"
server_key: "e3e86c3f0916178bce45d6b028d5d40c"
server_key_iv: "0a564fbe6eb853e750a1ed0b410dec62"
server_key_version: "encryptionkey@0"
I20260619 14:55:35.558944 7365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:35.568830 7365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:35.571020 7365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:35.571197 7365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:35.577606 7365 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41531
I20260619 14:55:35.577618 7432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41531 every 8 connection(s)
I20260619 14:55:35.578373 7365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:35.580484 7433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:35.584856 7433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e: Bootstrap starting.
I20260619 14:55:35.586719 7433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:35.587306 7433 log.cc:824] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:35.587365 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 7365
I20260619 14:55:35.587527 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:35.587973 19593 external_mini_cluster.cc:1428] Setting key c9c24615233c3da1e46ffc9a02fffe26
I20260619 14:55:35.589622 7433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e: No bootstrap required, opened a new log
I20260619 14:55:35.592736 7433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc5e2bec3ae4d05ad86896dad27db0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } }
I20260619 14:55:35.592960 7433 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:35.593019 7433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc5e2bec3ae4d05ad86896dad27db0e, State: Initialized, Role: FOLLOWER
I20260619 14:55:35.593393 7433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc5e2bec3ae4d05ad86896dad27db0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } }
I20260619 14:55:35.593501 7433 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:35.593582 7433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:35.593704 7433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:35.594470 7433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc5e2bec3ae4d05ad86896dad27db0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } }
I20260619 14:55:35.594761 7433 leader_election.cc:302] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dc5e2bec3ae4d05ad86896dad27db0e; no voters:
I20260619 14:55:35.595019 7433 leader_election.cc:288] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:35.595175 7438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:35.595458 7438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e [term 1 LEADER]: Becoming Leader. State: Replica: 3dc5e2bec3ae4d05ad86896dad27db0e, State: Running, Role: LEADER
I20260619 14:55:35.595749 7438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc5e2bec3ae4d05ad86896dad27db0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } }
I20260619 14:55:35.596141 7433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 14:55:35 dist-test-slave-3qdh krb5kdc[7352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:55:35.596819 7438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dc5e2bec3ae4d05ad86896dad27db0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc5e2bec3ae4d05ad86896dad27db0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } } }
I20260619 14:55:35.596935 7438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:35.597180 7438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dc5e2bec3ae4d05ad86896dad27db0e. Latest consensus state: current_term: 1 leader_uuid: "3dc5e2bec3ae4d05ad86896dad27db0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dc5e2bec3ae4d05ad86896dad27db0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41531 } } }
I20260619 14:55:35.597268 7438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:35.598147 7445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:35.601758 7445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:35.602433 7437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:35.589502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39925 (local address 127.0.0.1:41531)
0619 14:55:35.589897 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:35.589910 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:35.589939 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:55:35.590447 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:35.590460 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:35.590725 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:35.591009 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:35.591800 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:35.592816 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:35.594067 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:35.594512 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:35.594698 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:35.597552 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:35.597560 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:35.597577 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:35.597607 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:35.599234 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:35.599746 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:35.599751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:35.599756 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:35.599858 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:35.600241 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:35.600244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:35.600245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:35.600652 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:35.600831 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:35.601471 (+ 640us) server_negotiation.cc:299] Negotiation successful
0619 14:55:35.601661 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":113,"threads_started":1}
I20260619 14:55:35.605464 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:35.606647 7445 catalog_manager.cc:1347] Generated new cluster ID: d4e5284a0a3e4096bb0ddbc1a0b47aef
I20260619 14:55:35.606704 7445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:35.617252 7437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:35.608721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50449 (local address 127.0.0.1:41531)
0619 14:55:35.608813 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:35.608817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:35.609066 (+ 249us) server_negotiation.cc:407] Connection header received
0619 14:55:35.609255 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:35.609258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:35.609294 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:35.609337 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:35.610118 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:35.610809 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:35.611890 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:35.612214 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:35.612281 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:35.613941 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:35.613944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:35.613947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:35.613970 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:35.615524 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:35.616021 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:35.616025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:35.616026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:35.616094 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:35.616456 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:35.616459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:35.616460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:35.616618 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:35.616711 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:35.617047 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 14:55:35.617121 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:55:35.628125 7445 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:35.629230 7445 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:35.642496 7445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dc5e2bec3ae4d05ad86896dad27db0e: Generated new TSK 0
I20260619 14:55:35.643149 7445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:35.646473 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 7365
W20260619 14:55:35.651186 7350 connection.cc:535] client connection to 127.0.0.1:41531 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41531): Connection reset by peer (error 104)
2026-06-19T14:55:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1269 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:35 dist-test-slave-3qdh krb5kdc[7464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:35 dist-test-slave-3qdh krb5kdc[7464](info): set up 2 sockets
Jun 19 14:55:35 dist-test-slave-3qdh krb5kdc[7464](info): commencing operation
krb5kdc: starting...
W20260619 14:55:36.592290 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:36 dist-test-slave-3qdh krb5kdc[7464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:55:36.698212 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44209
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:36.792676 7477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:36.792861 7477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:36.792896 7477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:36.796355 7477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:36.796408 7477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:36.796424 7477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:36.796438 7477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:36.796450 7477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:36.800717 7477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44209
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:36.801440 7477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:36.802174 7477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:36.809393 7485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:36.809430 7482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:36.809468 7477 server_base.cc:1034] running on GCE node
W20260619 14:55:36.809430 7483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:36.810225 7477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:36.811551 7477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:36.814868 7477 hybrid_clock.cc:648] HybridClock initialized: now 1781880936814831 us; error 54 us; skew 500 ppm
Jun 19 14:55:36 dist-test-slave-3qdh krb5kdc[7464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:36.821295 7477 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:55:36.822043 7477 webserver.cc:460] Webserver started at http://127.0.0.1:41827/ using document root <none> and password file <none>
I20260619 14:55:36.822516 7477 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:36.822597 7477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:36.822767 7477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:36.824630 7477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "ff6536157eb54f35a3cfedaab10db9b3"
format_stamp: "Formatted at 2026-06-19 14:55:36 on dist-test-slave-3qdh"
server_key: "251e3fd81aceaa9be797d8f3f8edd0c4"
server_key_iv: "a28cd1e7f1c869e8a963c4348d133c4d"
server_key_version: "encryptionkey@0"
I20260619 14:55:36.825140 7477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "ff6536157eb54f35a3cfedaab10db9b3"
format_stamp: "Formatted at 2026-06-19 14:55:36 on dist-test-slave-3qdh"
server_key: "251e3fd81aceaa9be797d8f3f8edd0c4"
server_key_iv: "a28cd1e7f1c869e8a963c4348d133c4d"
server_key_version: "encryptionkey@0"
I20260619 14:55:36.828199 7477 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:55:36.830606 7492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:36.831698 7477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:36.831851 7477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "ff6536157eb54f35a3cfedaab10db9b3"
format_stamp: "Formatted at 2026-06-19 14:55:36 on dist-test-slave-3qdh"
server_key: "251e3fd81aceaa9be797d8f3f8edd0c4"
server_key_iv: "a28cd1e7f1c869e8a963c4348d133c4d"
server_key_version: "encryptionkey@0"
I20260619 14:55:36.831959 7477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:36.845396 7477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:36.848374 7477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:36.848584 7477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:36.855427 7477 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44209
I20260619 14:55:36.855435 7544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44209 every 8 connection(s)
I20260619 14:55:36.856201 7477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:36.858464 7545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:36.862957 7545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3: Bootstrap starting.
I20260619 14:55:36.863344 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 7477
I20260619 14:55:36.863510 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:36.863792 19593 external_mini_cluster.cc:1428] Setting key 0f3415f230e480b1cdbdf2d9d2c7faee
I20260619 14:55:36.865319 7545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:36.866003 7545 log.cc:824] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:36.867713 7545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3: No bootstrap required, opened a new log
I20260619 14:55:36.871541 7545 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6536157eb54f35a3cfedaab10db9b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44209 } }
I20260619 14:55:36.871743 7545 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:36.871815 7545 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6536157eb54f35a3cfedaab10db9b3, State: Initialized, Role: FOLLOWER
I20260619 14:55:36.872205 7548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:36.865004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48951 (local address 127.0.0.1:44209)
0619 14:55:36.865391 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:36.865428 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:36.865467 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:55:36.866025 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:36.866038 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:36.866430 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:36.866763 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:36.867053 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:36.867996 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:36.869240 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:36.869663 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:36.869772 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:36.870579 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:36.870584 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:36.870594 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:36.870903 (+ 309us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:36.870951 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:36.870972 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:36.871130 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:36.871344 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 14:55:36.871467 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":117,"threads_started":1}
I20260619 14:55:36.872221 7545 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6536157eb54f35a3cfedaab10db9b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44209 } }
I20260619 14:55:36.872471 7545 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:36.872538 7545 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:36.872615 7545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:36.873466 7545 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6536157eb54f35a3cfedaab10db9b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44209 } }
I20260619 14:55:36.873802 7545 leader_election.cc:302] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff6536157eb54f35a3cfedaab10db9b3; no voters:
I20260619 14:55:36.874107 7545 leader_election.cc:288] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:36.874210 7550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:36.874464 7550 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3 [term 1 LEADER]: Becoming Leader. State: Replica: ff6536157eb54f35a3cfedaab10db9b3, State: Running, Role: LEADER
I20260619 14:55:36.874830 7550 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6536157eb54f35a3cfedaab10db9b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44209 } }
I20260619 14:55:36.874984 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:36.875232 7545 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:36.876216 7552 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff6536157eb54f35a3cfedaab10db9b3. Latest consensus state: current_term: 1 leader_uuid: "ff6536157eb54f35a3cfedaab10db9b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6536157eb54f35a3cfedaab10db9b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44209 } } }
I20260619 14:55:36.876413 7552 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:36.877185 7551 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff6536157eb54f35a3cfedaab10db9b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6536157eb54f35a3cfedaab10db9b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44209 } } }
I20260619 14:55:36.877415 7551 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:36.877300 7564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:36.880788 7564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:36.881017 7548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:36.876928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45081 (local address 127.0.0.1:44209)
0619 14:55:36.877036 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:36.877041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:36.877506 (+ 465us) server_negotiation.cc:407] Connection header received
0619 14:55:36.877648 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:36.877657 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:36.877699 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:36.877791 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:36.878051 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:36.878748 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:36.879529 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:36.879764 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:36.879866 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:36.880596 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:36.880599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:36.880601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:36.880612 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:36.880660 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:36.880670 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:36.880762 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:36.880870 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 14:55:36.880897 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:55:36.886137 7564 catalog_manager.cc:1347] Generated new cluster ID: da0626ca5bad42089417bd6710525027
I20260619 14:55:36.886214 7564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:36.892588 7564 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:36.893306 7564 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:36.901077 7564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff6536157eb54f35a3cfedaab10db9b3: Generated new TSK 0
I20260619 14:55:36.901568 7564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:36.904712 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 7477
W20260619 14:55:36.909546 7462 connection.cc:535] client connection to 127.0.0.1:44209 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44209): Connection reset by peer (error 104)
2026-06-19T14:55:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1258 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:36 dist-test-slave-3qdh krb5kdc[7576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:36 dist-test-slave-3qdh krb5kdc[7576](info): set up 2 sockets
Jun 19 14:55:36 dist-test-slave-3qdh krb5kdc[7576](info): commencing operation
krb5kdc: starting...
W20260619 14:55:37.884397 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.961s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:37 dist-test-slave-3qdh krb5kdc[7576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:55:37.993037 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33385
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:38.092355 7589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:38.092561 7589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:38.092619 7589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:38.096254 7589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:38.096362 7589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:38.096400 7589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:38.096416 7589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:38.096431 7589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:38.100734 7589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33385
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:38.101468 7589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:38.102267 7589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:38.109817 7597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:38.109826 7594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:38.109966 7589 server_base.cc:1034] running on GCE node
W20260619 14:55:38.109979 7595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:38.110582 7589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:38.111531 7589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:38.112713 7589 hybrid_clock.cc:648] HybridClock initialized: now 1781880938112691 us; error 37 us; skew 500 ppm
Jun 19 14:55:38 dist-test-slave-3qdh krb5kdc[7576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:38.115249 7589 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:55:38.116024 7589 webserver.cc:460] Webserver started at http://127.0.0.1:43253/ using document root <none> and password file <none>
I20260619 14:55:38.116546 7589 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:38.116600 7589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:38.116758 7589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:38.118468 7589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "09bfc8cb97b34add88499393fe650167"
format_stamp: "Formatted at 2026-06-19 14:55:38 on dist-test-slave-3qdh"
server_key: "4b4dbe0c205caf6fcaeee0519d6895f3"
server_key_iv: "d5fdecac892f688aa2518ddec2c6e6ab"
server_key_version: "encryptionkey@0"
I20260619 14:55:38.118883 7589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "09bfc8cb97b34add88499393fe650167"
format_stamp: "Formatted at 2026-06-19 14:55:38 on dist-test-slave-3qdh"
server_key: "4b4dbe0c205caf6fcaeee0519d6895f3"
server_key_iv: "d5fdecac892f688aa2518ddec2c6e6ab"
server_key_version: "encryptionkey@0"
I20260619 14:55:38.121918 7589 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:55:38.123943 7604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:38.125036 7589 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:38.125156 7589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "09bfc8cb97b34add88499393fe650167"
format_stamp: "Formatted at 2026-06-19 14:55:38 on dist-test-slave-3qdh"
server_key: "4b4dbe0c205caf6fcaeee0519d6895f3"
server_key_iv: "d5fdecac892f688aa2518ddec2c6e6ab"
server_key_version: "encryptionkey@0"
I20260619 14:55:38.125242 7589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:38.129249 7589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:38.131600 7589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:38.131783 7589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:38.137972 7589 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33385
I20260619 14:55:38.138000 7656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33385 every 8 connection(s)
I20260619 14:55:38.138710 7589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:38.140991 7657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:38.145687 7657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167: Bootstrap starting.
I20260619 14:55:38.147590 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 7589
I20260619 14:55:38.147722 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:38.147914 7657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:38.148067 19593 external_mini_cluster.cc:1428] Setting key 616794260a768545e0c4ca7bb742bfd9
I20260619 14:55:38.148694 7657 log.cc:824] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:38.150750 7657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167: No bootstrap required, opened a new log
I20260619 14:55:38.153003 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:38.149099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48445 (local address 127.0.0.1:33385)
0619 14:55:38.149434 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:38.149477 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:38.149512 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:55:38.150259 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:38.150278 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:38.150601 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:38.150942 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:38.151776 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:38.151781 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:38.151794 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:38.151805 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:38.151861 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:38.151884 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:38.151990 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:38.152068 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 14:55:38.152205 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":88,"threads_started":1}
I20260619 14:55:38.154953 7657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bfc8cb97b34add88499393fe650167" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33385 } }
I20260619 14:55:38.155203 7657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:38.155288 7657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09bfc8cb97b34add88499393fe650167, State: Initialized, Role: FOLLOWER
I20260619 14:55:38.155604 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:38.155665 7657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bfc8cb97b34add88499393fe650167" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33385 } }
I20260619 14:55:38.155833 7657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:38.155893 7657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:38.155977 7657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:38.156785 7657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bfc8cb97b34add88499393fe650167" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33385 } }
I20260619 14:55:38.157066 7657 leader_election.cc:302] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09bfc8cb97b34add88499393fe650167; no voters:
I20260619 14:55:38.157337 7657 leader_election.cc:288] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:38.157442 7666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:38.157655 7666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167 [term 1 LEADER]: Becoming Leader. State: Replica: 09bfc8cb97b34add88499393fe650167, State: Running, Role: LEADER
I20260619 14:55:38.158003 7666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bfc8cb97b34add88499393fe650167" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33385 } }
I20260619 14:55:38.158412 7657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:38.159173 7666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09bfc8cb97b34add88499393fe650167. Latest consensus state: current_term: 1 leader_uuid: "09bfc8cb97b34add88499393fe650167" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bfc8cb97b34add88499393fe650167" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33385 } } }
I20260619 14:55:38.159291 7666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:38.159482 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:38.157789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37993 (local address 127.0.0.1:33385)
0619 14:55:38.157887 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:38.157891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:38.158017 (+ 126us) server_negotiation.cc:407] Connection header received
0619 14:55:38.158175 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:38.158178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:38.158209 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:38.158266 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:38.159081 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:38.159083 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:38.159086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:38.159097 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:38.159117 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:38.159126 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:38.159173 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:38.159336 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 14:55:38.159369 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 14:55:38.159622 7676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:38.159755 7667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09bfc8cb97b34add88499393fe650167" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bfc8cb97b34add88499393fe650167" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33385 } } }
I20260619 14:55:38.159960 7667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:38.162709 7676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:38.167405 7676 catalog_manager.cc:1347] Generated new cluster ID: d0d00206df9e46f3a14bb6429e5a3791
I20260619 14:55:38.167460 7676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:38.181113 7676 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:38.181377 7676 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:38.186077 7676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09bfc8cb97b34add88499393fe650167: Generated new TSK 0
I20260619 14:55:38.186704 7676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:38.193522 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 7589
2026-06-19T14:55:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1288 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:38 dist-test-slave-3qdh krb5kdc[7688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:38 dist-test-slave-3qdh krb5kdc[7688](info): set up 2 sockets
Jun 19 14:55:38 dist-test-slave-3qdh krb5kdc[7688](info): commencing operation
krb5kdc: starting...
W20260619 14:55:39.168094 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:39 dist-test-slave-3qdh krb5kdc[7688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:55:39.274672 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33571
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:39.375598 7701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:39.375854 7701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:39.375931 7701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:39.379508 7701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:39.379583 7701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:39.379618 7701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:39.379670 7701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:39.379704 7701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:39.384562 7701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:39.385347 7701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:39.386194 7701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:39.393754 7709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:39.393762 7706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:39.393808 7701 server_base.cc:1034] running on GCE node
W20260619 14:55:39.393774 7707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:39.394529 7701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:39.395460 7701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:39.396624 7701 hybrid_clock.cc:648] HybridClock initialized: now 1781880939396605 us; error 29 us; skew 500 ppm
Jun 19 14:55:39 dist-test-slave-3qdh krb5kdc[7688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:39.399164 7701 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:55:39.399926 7701 webserver.cc:460] Webserver started at http://127.0.0.1:36595/ using document root <none> and password file <none>
I20260619 14:55:39.400486 7701 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:39.400574 7701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:39.400790 7701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:39.402575 7701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "ead96b7131034d518d67fbb96adc7c08"
format_stamp: "Formatted at 2026-06-19 14:55:39 on dist-test-slave-3qdh"
server_key: "0cb7722681c35d6db8ed4c2a871248b6"
server_key_iv: "2def49dfbadeb7edad8fda13b82b8098"
server_key_version: "encryptionkey@0"
I20260619 14:55:39.403060 7701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "ead96b7131034d518d67fbb96adc7c08"
format_stamp: "Formatted at 2026-06-19 14:55:39 on dist-test-slave-3qdh"
server_key: "0cb7722681c35d6db8ed4c2a871248b6"
server_key_iv: "2def49dfbadeb7edad8fda13b82b8098"
server_key_version: "encryptionkey@0"
I20260619 14:55:39.406481 7701 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260619 14:55:39.408649 7716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:39.409770 7701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260619 14:55:39.409880 7701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "ead96b7131034d518d67fbb96adc7c08"
format_stamp: "Formatted at 2026-06-19 14:55:39 on dist-test-slave-3qdh"
server_key: "0cb7722681c35d6db8ed4c2a871248b6"
server_key_iv: "2def49dfbadeb7edad8fda13b82b8098"
server_key_version: "encryptionkey@0"
I20260619 14:55:39.409962 7701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:39.427393 7701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:39.430359 7701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:39.430502 7701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:39.436697 7701 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33571
I20260619 14:55:39.436718 7768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33571 every 8 connection(s)
I20260619 14:55:39.437641 7701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:39.440042 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 7701
I20260619 14:55:39.440186 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:39.440337 7769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:39.440508 19593 external_mini_cluster.cc:1428] Setting key 269d580cabe9774792c76600ad38629c
I20260619 14:55:39.447167 7769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08: Bootstrap starting.
I20260619 14:55:39.449566 7769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08: Neither blocks nor log segments found. Creating new log.
Jun 19 14:55:39 dist-test-slave-3qdh krb5kdc[7688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:55:39.450444 7769 log.cc:824] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:39.452422 7769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08: No bootstrap required, opened a new log
I20260619 14:55:39.455574 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:39.441790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49571 (local address 127.0.0.1:33571)
0619 14:55:39.442223 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:39.442239 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:39.442277 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:55:39.442937 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:39.442958 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:39.443366 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:39.443701 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:39.444351 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:39.445772 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:39.446782 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:39.447472 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:39.447632 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:39.450472 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:39.450477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:39.450488 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:39.450550 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:39.452902 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:39.453389 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:39.453394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:39.453398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:39.453517 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:39.453824 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:39.453826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:39.453827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:39.454172 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:39.454329 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:39.454658 (+ 329us) server_negotiation.cc:299] Negotiation successful
0619 14:55:39.454933 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":123,"threads_started":1}
I20260619 14:55:39.456187 7769 raft_consensus.cc:348] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ead96b7131034d518d67fbb96adc7c08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33571 } }
I20260619 14:55:39.456493 7769 raft_consensus.cc:374] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:39.456557 7769 raft_consensus.cc:729] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ead96b7131034d518d67fbb96adc7c08, State: Initialized, Role: FOLLOWER
I20260619 14:55:39.456946 7769 consensus_queue.cc:260] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ead96b7131034d518d67fbb96adc7c08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33571 } }
I20260619 14:55:39.457082 7769 raft_consensus.cc:388] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:39.457197 7769 raft_consensus.cc:482] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:39.457343 7769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:39.458316 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:39.458365 7769 raft_consensus.cc:504] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ead96b7131034d518d67fbb96adc7c08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33571 } }
I20260619 14:55:39.458671 7769 leader_election.cc:302] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ead96b7131034d518d67fbb96adc7c08; no voters:
I20260619 14:55:39.458969 7769 leader_election.cc:288] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:39.459112 7774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:39.459385 7774 raft_consensus.cc:686] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08 [term 1 LEADER]: Becoming Leader. State: Replica: ead96b7131034d518d67fbb96adc7c08, State: Running, Role: LEADER
I20260619 14:55:39.459702 7774 consensus_queue.cc:237] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ead96b7131034d518d67fbb96adc7c08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33571 } }
I20260619 14:55:39.460142 7769 sys_catalog.cc:564] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:39.461308 7775 sys_catalog.cc:455] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ead96b7131034d518d67fbb96adc7c08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ead96b7131034d518d67fbb96adc7c08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33571 } } }
I20260619 14:55:39.461419 7775 sys_catalog.cc:458] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:39.461838 7788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:39.461812 7776 sys_catalog.cc:455] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ead96b7131034d518d67fbb96adc7c08. Latest consensus state: current_term: 1 leader_uuid: "ead96b7131034d518d67fbb96adc7c08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ead96b7131034d518d67fbb96adc7c08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33571 } } }
I20260619 14:55:39.462400 7776 sys_catalog.cc:458] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:39.465004 7788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:39.469913 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:39.460432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42967 (local address 127.0.0.1:33571)
0619 14:55:39.460516 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:39.460520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:39.462665 (+ 2145us) server_negotiation.cc:407] Connection header received
0619 14:55:39.462850 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:39.462853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:39.462901 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:39.462979 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:39.463632 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:39.464118 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:39.464971 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:39.465244 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:39.465351 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:39.466582 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:39.466585 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:39.466587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:39.466617 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:39.468027 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:39.468527 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:39.468531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:39.468533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:39.468638 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:39.468959 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:39.468963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:39.468964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:39.469211 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:39.469317 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:39.469641 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 14:55:39.469748 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260619 14:55:39.471014 7788 catalog_manager.cc:1347] Generated new cluster ID: c4f4ec5e4fd341f2901ffefd0dc001c8
I20260619 14:55:39.471082 7788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:39.507365 7788 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:39.508281 7788 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:39.518939 7788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ead96b7131034d518d67fbb96adc7c08: Generated new TSK 0
I20260619 14:55:39.519656 7788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:39.526705 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 7701
2026-06-19T14:55:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:39 dist-test-slave-3qdh krb5kdc[7800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:39 dist-test-slave-3qdh krb5kdc[7800](info): set up 2 sockets
Jun 19 14:55:39 dist-test-slave-3qdh krb5kdc[7800](info): commencing operation
krb5kdc: starting...
W20260619 14:55:40.476871 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:40 dist-test-slave-3qdh krb5kdc[7800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:55:40.594949 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35079
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:40.689838 7813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:40.690038 7813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:40.690073 7813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:40.693782 7813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:40.693831 7813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:40.693850 7813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:40.693867 7813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:40.693912 7813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:40.698345 7813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:40.699149 7813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:40.700008 7813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:40.707809 7819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:40.707948 7813 server_base.cc:1034] running on GCE node
W20260619 14:55:40.707849 7818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:40.707892 7821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:40.708632 7813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:40.709627 7813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:40.710786 7813 hybrid_clock.cc:648] HybridClock initialized: now 1781880940710770 us; error 28 us; skew 500 ppm
Jun 19 14:55:40 dist-test-slave-3qdh krb5kdc[7800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880940, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:40.713442 7813 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:55:40.714195 7813 webserver.cc:460] Webserver started at http://127.0.0.1:45361/ using document root <none> and password file <none>
I20260619 14:55:40.714711 7813 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:40.714807 7813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:40.715029 7813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:40.716974 7813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "0e10fae8284441b897b9bc8ba529af04"
format_stamp: "Formatted at 2026-06-19 14:55:40 on dist-test-slave-3qdh"
server_key: "f5a7b10474253ddfbcf64b129085fa0f"
server_key_iv: "9f3781267baf29a710315a7e3e67be54"
server_key_version: "encryptionkey@0"
I20260619 14:55:40.717486 7813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "0e10fae8284441b897b9bc8ba529af04"
format_stamp: "Formatted at 2026-06-19 14:55:40 on dist-test-slave-3qdh"
server_key: "f5a7b10474253ddfbcf64b129085fa0f"
server_key_iv: "9f3781267baf29a710315a7e3e67be54"
server_key_version: "encryptionkey@0"
I20260619 14:55:40.720830 7813 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:55:40.723066 7828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:40.724152 7813 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:40.724340 7813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "0e10fae8284441b897b9bc8ba529af04"
format_stamp: "Formatted at 2026-06-19 14:55:40 on dist-test-slave-3qdh"
server_key: "f5a7b10474253ddfbcf64b129085fa0f"
server_key_iv: "9f3781267baf29a710315a7e3e67be54"
server_key_version: "encryptionkey@0"
I20260619 14:55:40.724463 7813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:40.754031 7813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:40.756639 7813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:40.756856 7813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:40.763209 7813 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35079
I20260619 14:55:40.763221 7880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35079 every 8 connection(s)
I20260619 14:55:40.764096 7813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:40.766479 7881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:40.769779 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 7813
I20260619 14:55:40.769930 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:40.770217 19593 external_mini_cluster.cc:1428] Setting key df8d9b2e5e0f17f596dc6138baafd025
I20260619 14:55:40.771835 7881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04: Bootstrap starting.
I20260619 14:55:40.774183 7881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:40.774962 7881 log.cc:824] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:40.776868 7881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04: No bootstrap required, opened a new log
I20260619 14:55:40.779155 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:40.771406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45249 (local address 127.0.0.1:35079)
0619 14:55:40.771755 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:40.771771 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:40.771805 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:55:40.772473 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:40.772491 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:40.772862 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:40.773161 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:40.773529 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:40.774960 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:40.775797 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:40.776423 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:40.776568 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:40.777278 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:40.777283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:40.777297 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:40.777709 (+ 412us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:40.777781 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:40.777808 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:40.777912 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:40.777994 (+ 82us) server_negotiation.cc:299] Negotiation successful
0619 14:55:40.778163 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":90,"threads_started":1}
I20260619 14:55:40.780232 7881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e10fae8284441b897b9bc8ba529af04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35079 } }
I20260619 14:55:40.780469 7881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:40.780520 7881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e10fae8284441b897b9bc8ba529af04, State: Initialized, Role: FOLLOWER
I20260619 14:55:40.780901 7881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e10fae8284441b897b9bc8ba529af04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35079 } }
I20260619 14:55:40.781040 7881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:40.781101 7881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:40.781189 7881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:40.781629 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:40.782037 7881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e10fae8284441b897b9bc8ba529af04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35079 } }
I20260619 14:55:40.782349 7881 leader_election.cc:302] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e10fae8284441b897b9bc8ba529af04; no voters:
I20260619 14:55:40.782594 7881 leader_election.cc:288] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:40.782701 7886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:40.782900 7886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04 [term 1 LEADER]: Becoming Leader. State: Replica: 0e10fae8284441b897b9bc8ba529af04, State: Running, Role: LEADER
I20260619 14:55:40.783228 7886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e10fae8284441b897b9bc8ba529af04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35079 } }
I20260619 14:55:40.783690 7881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:40.784267 7886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e10fae8284441b897b9bc8ba529af04. Latest consensus state: current_term: 1 leader_uuid: "0e10fae8284441b897b9bc8ba529af04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e10fae8284441b897b9bc8ba529af04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35079 } } }
I20260619 14:55:40.784456 7886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:40.784734 7892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e10fae8284441b897b9bc8ba529af04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e10fae8284441b897b9bc8ba529af04" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35079 } } }
I20260619 14:55:40.784883 7898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:40.784897 7892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:40.788934 7898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:40.790508 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:40.785870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38623 (local address 127.0.0.1:35079)
0619 14:55:40.785984 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:40.785988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:40.786154 (+ 166us) server_negotiation.cc:407] Connection header received
0619 14:55:40.786280 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:40.786282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:40.786326 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:40.786394 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:40.786663 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:40.787304 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:40.788921 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:40.789249 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:40.789345 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:40.790116 (+ 771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:40.790119 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:40.790121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:40.790132 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:40.790155 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:40.790167 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:40.790231 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:40.790351 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 14:55:40.790395 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:55:40.795125 7898 catalog_manager.cc:1347] Generated new cluster ID: f05e261bd02340af9ba4a5a023bf43db
I20260619 14:55:40.795222 7898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:40.803007 7898 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:40.804067 7898 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:40.813925 7898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e10fae8284441b897b9bc8ba529af04: Generated new TSK 0
I20260619 14:55:40.814671 7898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:40.824137 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 7813
2026-06-19T14:55:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:40 dist-test-slave-3qdh krb5kdc[7912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:40 dist-test-slave-3qdh krb5kdc[7912](info): set up 2 sockets
Jun 19 14:55:40 dist-test-slave-3qdh krb5kdc[7912](info): commencing operation
krb5kdc: starting...
W20260619 14:55:41.792655 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:41 dist-test-slave-3qdh krb5kdc[7912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:55:41.898353 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39191
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:41.993340 7925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:41.993567 7925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:41.993607 7925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:41.997097 7925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:41.997139 7925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:41.997155 7925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:41.997215 7925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:41.997246 7925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:42.001758 7925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:42.002578 7925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:42.003381 7925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:42.011144 7930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:42.011214 7925 server_base.cc:1034] running on GCE node
W20260619 14:55:42.011224 7931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:42.011240 7933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:42.011838 7925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:42.012840 7925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:42.014055 7925 hybrid_clock.cc:648] HybridClock initialized: now 1781880942014026 us; error 44 us; skew 500 ppm
Jun 19 14:55:42 dist-test-slave-3qdh krb5kdc[7912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:42.021952 7925 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:55:42.022701 7925 webserver.cc:460] Webserver started at http://127.0.0.1:44211/ using document root <none> and password file <none>
I20260619 14:55:42.023193 7925 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:42.023280 7925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:42.023526 7925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:42.025427 7925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "ca4dbebd3ce84917af28b50e07f1865c"
format_stamp: "Formatted at 2026-06-19 14:55:42 on dist-test-slave-3qdh"
server_key: "070cc2f1ab68df17e1ed0ecf0b2c210e"
server_key_iv: "5c1160bd39fcdd953528aeaa51be543f"
server_key_version: "encryptionkey@0"
I20260619 14:55:42.025925 7925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "ca4dbebd3ce84917af28b50e07f1865c"
format_stamp: "Formatted at 2026-06-19 14:55:42 on dist-test-slave-3qdh"
server_key: "070cc2f1ab68df17e1ed0ecf0b2c210e"
server_key_iv: "5c1160bd39fcdd953528aeaa51be543f"
server_key_version: "encryptionkey@0"
I20260619 14:55:42.029109 7925 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:55:42.031333 7940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:42.032505 7925 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:55:42.032632 7925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "ca4dbebd3ce84917af28b50e07f1865c"
format_stamp: "Formatted at 2026-06-19 14:55:42 on dist-test-slave-3qdh"
server_key: "070cc2f1ab68df17e1ed0ecf0b2c210e"
server_key_iv: "5c1160bd39fcdd953528aeaa51be543f"
server_key_version: "encryptionkey@0"
I20260619 14:55:42.032730 7925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:42.036675 7925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:42.039427 7925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:42.039602 7925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:42.046696 7925 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39191
I20260619 14:55:42.046700 7992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39191 every 8 connection(s)
I20260619 14:55:42.047472 7925 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:42.049966 7993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:42.052862 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 7925
I20260619 14:55:42.052976 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:42.053244 19593 external_mini_cluster.cc:1428] Setting key 2d26e8db8142f53dcbc724e521060b24
I20260619 14:55:42.055233 7993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c: Bootstrap starting.
I20260619 14:55:42.057505 7993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:42.057963 7996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:42.054484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33617 (local address 127.0.0.1:39191)
0619 14:55:42.054901 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:42.054915 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:42.054948 (+ 33us) server_negotiation.cc:407] Connection header received
0619 14:55:42.055586 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:42.055603 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:42.055870 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:42.056137 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:42.056874 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:42.056878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:42.056888 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:42.056897 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:42.056956 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:42.056974 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:42.057054 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:42.057182 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 14:55:42.057330 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":145,"threads_started":1}
I20260619 14:55:42.058171 7993 log.cc:824] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:42.059818 7993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c: No bootstrap required, opened a new log
I20260619 14:55:42.060386 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:42.064021 7996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:42.062510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39319 (local address 127.0.0.1:39191)
0619 14:55:42.062624 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:42.062628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:42.062796 (+ 168us) server_negotiation.cc:407] Connection header received
0619 14:55:42.062913 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:42.062915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:42.062939 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:42.062999 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:42.063630 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:42.063633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:42.063634 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:42.063644 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:42.063663 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:42.063673 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:42.063737 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:42.063852 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 14:55:42.063878 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:55:42.064291 7993 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca4dbebd3ce84917af28b50e07f1865c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39191 } }
I20260619 14:55:42.064569 7993 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:42.064620 7993 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca4dbebd3ce84917af28b50e07f1865c, State: Initialized, Role: FOLLOWER
I20260619 14:55:42.065016 7993 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca4dbebd3ce84917af28b50e07f1865c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39191 } }
I20260619 14:55:42.065115 7993 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:42.065155 7993 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:42.065212 7993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:42.065836 7993 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca4dbebd3ce84917af28b50e07f1865c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39191 } }
I20260619 14:55:42.066062 7993 leader_election.cc:302] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca4dbebd3ce84917af28b50e07f1865c; no voters:
I20260619 14:55:42.066249 7993 leader_election.cc:288] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:42.066341 8003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:42.066533 8003 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c [term 1 LEADER]: Becoming Leader. State: Replica: ca4dbebd3ce84917af28b50e07f1865c, State: Running, Role: LEADER
I20260619 14:55:42.066816 8003 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca4dbebd3ce84917af28b50e07f1865c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39191 } }
I20260619 14:55:42.067188 7993 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:42.068377 8005 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca4dbebd3ce84917af28b50e07f1865c. Latest consensus state: current_term: 1 leader_uuid: "ca4dbebd3ce84917af28b50e07f1865c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca4dbebd3ce84917af28b50e07f1865c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39191 } } }
I20260619 14:55:42.068404 8004 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca4dbebd3ce84917af28b50e07f1865c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca4dbebd3ce84917af28b50e07f1865c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39191 } } }
I20260619 14:55:42.068487 8005 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:42.068495 8004 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:42.069440 8013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:42.071619 8013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:42.075680 8013 catalog_manager.cc:1347] Generated new cluster ID: 22711ea068c34637a464a738235ca2f3
I20260619 14:55:42.075728 8013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:42.090229 8013 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:42.090487 8013 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:42.101459 8013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca4dbebd3ce84917af28b50e07f1865c: Generated new TSK 0
I20260619 14:55:42.101948 8013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:42.107226 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 7925
2026-06-19T14:55:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1283 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:42 dist-test-slave-3qdh krb5kdc[8024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:42 dist-test-slave-3qdh krb5kdc[8024](info): set up 2 sockets
Jun 19 14:55:42 dist-test-slave-3qdh krb5kdc[8024](info): commencing operation
krb5kdc: starting...
W20260619 14:55:43.060439 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:43 dist-test-slave-3qdh krb5kdc[8024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:55:43.167060 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:43.264943 8037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:43.265130 8037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:43.265169 8037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:43.268560 8037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:43.268602 8037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:43.268620 8037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:43.268636 8037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:43.268651 8037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:43.273001 8037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:43.273718 8037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:43.274531 8037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:43.281522 8042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:43.281558 8045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:43.281538 8043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:43.281656 8037 server_base.cc:1034] running on GCE node
I20260619 14:55:43.282187 8037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:43.283113 8037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:43.284344 8037 hybrid_clock.cc:648] HybridClock initialized: now 1781880943284286 us; error 27 us; skew 500 ppm
Jun 19 14:55:43 dist-test-slave-3qdh krb5kdc[8024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880943, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:43.286662 8037 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:55:43.287386 8037 webserver.cc:460] Webserver started at http://127.0.0.1:38829/ using document root <none> and password file <none>
I20260619 14:55:43.287899 8037 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:43.287987 8037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:43.288208 8037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:43.289947 8037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "3341fba145e64640a604231267d1ee7f"
format_stamp: "Formatted at 2026-06-19 14:55:43 on dist-test-slave-3qdh"
server_key: "16f78ff5967ddfe48840e4f6f1cda058"
server_key_iv: "349c6e4925439d2d61564a221e160b8b"
server_key_version: "encryptionkey@0"
I20260619 14:55:43.290436 8037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "3341fba145e64640a604231267d1ee7f"
format_stamp: "Formatted at 2026-06-19 14:55:43 on dist-test-slave-3qdh"
server_key: "16f78ff5967ddfe48840e4f6f1cda058"
server_key_iv: "349c6e4925439d2d61564a221e160b8b"
server_key_version: "encryptionkey@0"
I20260619 14:55:43.293602 8037 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:55:43.295682 8052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:43.296747 8037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:55:43.296864 8037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "3341fba145e64640a604231267d1ee7f"
format_stamp: "Formatted at 2026-06-19 14:55:43 on dist-test-slave-3qdh"
server_key: "16f78ff5967ddfe48840e4f6f1cda058"
server_key_iv: "349c6e4925439d2d61564a221e160b8b"
server_key_version: "encryptionkey@0"
I20260619 14:55:43.296957 8037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:43.313661 8037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:43.315963 8037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:43.316136 8037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:43.322618 8037 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39855
I20260619 14:55:43.322643 8104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39855 every 8 connection(s)
I20260619 14:55:43.323374 8037 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:43.325779 8105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:43.330497 8105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f: Bootstrap starting.
I20260619 14:55:43.331960 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 8037
I20260619 14:55:43.332050 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:43.332351 19593 external_mini_cluster.cc:1428] Setting key 3cdda5dfbc57f5cea26acedcdbe78a72
I20260619 14:55:43.332752 8105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:43.333514 8105 log.cc:824] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:43.335176 8105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f: No bootstrap required, opened a new log
I20260619 14:55:43.338044 8105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3341fba145e64640a604231267d1ee7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39855 } }
I20260619 14:55:43.338240 8105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:43.338308 8105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3341fba145e64640a604231267d1ee7f, State: Initialized, Role: FOLLOWER
I20260619 14:55:43.338716 8105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3341fba145e64640a604231267d1ee7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39855 } }
I20260619 14:55:43.338829 8105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:43.338930 8105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:43.339042 8105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:43.339847 8105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3341fba145e64640a604231267d1ee7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39855 } }
Jun 19 14:55:43 dist-test-slave-3qdh krb5kdc[8024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880943, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:55:43.340167 8105 leader_election.cc:302] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3341fba145e64640a604231267d1ee7f; no voters:
I20260619 14:55:43.340472 8105 leader_election.cc:288] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:43.340582 8110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:43.340778 8110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f [term 1 LEADER]: Becoming Leader. State: Replica: 3341fba145e64640a604231267d1ee7f, State: Running, Role: LEADER
I20260619 14:55:43.341090 8110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3341fba145e64640a604231267d1ee7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39855 } }
I20260619 14:55:43.341496 8105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:43.342567 8111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3341fba145e64640a604231267d1ee7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3341fba145e64640a604231267d1ee7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39855 } } }
I20260619 14:55:43.342669 8111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:43.342959 8112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3341fba145e64640a604231267d1ee7f. Latest consensus state: current_term: 1 leader_uuid: "3341fba145e64640a604231267d1ee7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3341fba145e64640a604231267d1ee7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39855 } } }
I20260619 14:55:43.343065 8112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:43.345480 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:43.333298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:58259 (local address 127.0.0.1:39855)
0619 14:55:43.333748 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:43.333763 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:43.333794 (+ 31us) server_negotiation.cc:407] Connection header received
0619 14:55:43.334526 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:43.334543 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:43.334843 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:43.335136 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:43.335868 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:43.336814 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:43.337571 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:43.337964 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:43.338111 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:43.340824 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:43.340831 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:43.340847 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:43.340875 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:43.342604 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:43.343224 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:43.343231 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:43.343237 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:43.343343 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:43.343762 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:43.343765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:43.343767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:43.344121 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:43.344383 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:43.344671 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 14:55:43.344914 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":113,"threads_started":1}
W20260619 14:55:43.346001 8125 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:55:43.346067 8125 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:55:43.346140 8120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:43.346860 8120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:43.348001 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:43.351951 8120 catalog_manager.cc:1347] Generated new cluster ID: 8041f7a546bf41dd9932964b35810af9
I20260619 14:55:43.351999 8120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:43.357564 8108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:43.350214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:41303 (local address 127.0.0.1:39855)
0619 14:55:43.350334 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:43.350338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:43.350475 (+ 137us) server_negotiation.cc:407] Connection header received
0619 14:55:43.350596 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:43.350599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:43.350650 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:43.350735 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:43.351508 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:43.352154 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:43.352930 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:43.353153 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:43.353259 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:43.354575 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:43.354578 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:43.354581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:43.354603 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:43.355929 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:43.356386 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:43.356389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:43.356391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:43.356461 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:43.356764 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:43.356767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:43.356769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:43.356914 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:43.357053 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:43.357319 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 14:55:43.357400 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:55:43.365576 8120 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:43.366370 8120 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:43.372916 8120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3341fba145e64640a604231267d1ee7f: Generated new TSK 0
I20260619 14:55:43.373535 8120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:43.382380 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 8037
W20260619 14:55:43.387571 8022 connection.cc:535] client connection to 127.0.0.1:39855 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39855): Connection reset by peer (error 104)
2026-06-19T14:55:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1275 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:43 dist-test-slave-3qdh krb5kdc[8136](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:43 dist-test-slave-3qdh krb5kdc[8136](info): set up 2 sockets
Jun 19 14:55:43 dist-test-slave-3qdh krb5kdc[8136](info): commencing operation
krb5kdc: starting...
W20260619 14:55:44.344040 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:44 dist-test-slave-3qdh krb5kdc[8136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:55:44.445323 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39021
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:44.539825 8149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:44.540041 8149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:44.540102 8149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:44.543360 8149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:44.543430 8149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:44.543463 8149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:44.543507 8149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:44.543550 8149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:44.547932 8149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:44.548774 8149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:44.549567 8149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:44.556532 8157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:44.556526 8155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:44.556617 8149 server_base.cc:1034] running on GCE node
W20260619 14:55:44.556532 8154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:44.557202 8149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:44.557988 8149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:44.559144 8149 hybrid_clock.cc:648] HybridClock initialized: now 1781880944559118 us; error 35 us; skew 500 ppm
Jun 19 14:55:44 dist-test-slave-3qdh krb5kdc[8136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880944, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:44.561441 8149 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:55:44.562067 8149 webserver.cc:460] Webserver started at http://127.0.0.1:45897/ using document root <none> and password file <none>
I20260619 14:55:44.562470 8149 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:44.562516 8149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:44.562645 8149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:44.564225 8149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "3ee4fb6b9a7747dd94b6e1f56682e039"
format_stamp: "Formatted at 2026-06-19 14:55:44 on dist-test-slave-3qdh"
server_key: "d6728a281a2389fbd178d2a1be32e3e4"
server_key_iv: "12f0c702d772929c3cdcf3295fd291f1"
server_key_version: "encryptionkey@0"
I20260619 14:55:44.564754 8149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "3ee4fb6b9a7747dd94b6e1f56682e039"
format_stamp: "Formatted at 2026-06-19 14:55:44 on dist-test-slave-3qdh"
server_key: "d6728a281a2389fbd178d2a1be32e3e4"
server_key_iv: "12f0c702d772929c3cdcf3295fd291f1"
server_key_version: "encryptionkey@0"
I20260619 14:55:44.567654 8149 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:55:44.569736 8164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:44.570690 8149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:55:44.570808 8149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "3ee4fb6b9a7747dd94b6e1f56682e039"
format_stamp: "Formatted at 2026-06-19 14:55:44 on dist-test-slave-3qdh"
server_key: "d6728a281a2389fbd178d2a1be32e3e4"
server_key_iv: "12f0c702d772929c3cdcf3295fd291f1"
server_key_version: "encryptionkey@0"
I20260619 14:55:44.570901 8149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:44.584661 8149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:44.587245 8149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:44.587414 8149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:44.593256 8149 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39021
I20260619 14:55:44.593269 8216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39021 every 8 connection(s)
I20260619 14:55:44.594079 8149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:44.596414 8217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:44.599962 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 8149
I20260619 14:55:44.600071 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:44.600364 19593 external_mini_cluster.cc:1428] Setting key fc58a0023009a3d1fb52f88b9418c9ce
I20260619 14:55:44.601459 8217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039: Bootstrap starting.
I20260619 14:55:44.603461 8217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:44.604133 8217 log.cc:824] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:44.605959 8217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039: No bootstrap required, opened a new log
I20260619 14:55:44.608057 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:44.601532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:50895 (local address 127.0.0.1:39021)
0619 14:55:44.601893 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:44.601907 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:44.601936 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:55:44.602470 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:44.602483 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:44.602754 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:44.602972 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:44.603275 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:44.604262 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:44.605379 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:44.605759 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:44.605858 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:44.606551 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:44.606555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:44.606565 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:44.606868 (+ 303us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:44.606929 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:44.606949 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:44.607060 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:44.607307 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 14:55:44.607434 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":103,"threads_started":1}
I20260619 14:55:44.609462 8217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee4fb6b9a7747dd94b6e1f56682e039" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39021 } }
I20260619 14:55:44.609715 8217 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:44.609771 8217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ee4fb6b9a7747dd94b6e1f56682e039, State: Initialized, Role: FOLLOWER
I20260619 14:55:44.610184 8217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee4fb6b9a7747dd94b6e1f56682e039" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39021 } }
I20260619 14:55:44.610317 8217 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:44.610370 8217 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:44.610455 8217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:44.610612 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:44.611203 8217 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee4fb6b9a7747dd94b6e1f56682e039" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39021 } }
I20260619 14:55:44.611486 8217 leader_election.cc:302] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ee4fb6b9a7747dd94b6e1f56682e039; no voters:
I20260619 14:55:44.611759 8217 leader_election.cc:288] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:44.611807 8225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:44.611994 8225 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039 [term 1 LEADER]: Becoming Leader. State: Replica: 3ee4fb6b9a7747dd94b6e1f56682e039, State: Running, Role: LEADER
I20260619 14:55:44.612321 8225 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee4fb6b9a7747dd94b6e1f56682e039" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39021 } }
I20260619 14:55:44.612852 8217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:44.613389 8225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ee4fb6b9a7747dd94b6e1f56682e039" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee4fb6b9a7747dd94b6e1f56682e039" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39021 } } }
I20260619 14:55:44.613502 8225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:44.613888 8233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:44.614557 8227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ee4fb6b9a7747dd94b6e1f56682e039. Latest consensus state: current_term: 1 leader_uuid: "3ee4fb6b9a7747dd94b6e1f56682e039" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ee4fb6b9a7747dd94b6e1f56682e039" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39021 } } }
I20260619 14:55:44.614693 8227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:44.616693 8233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:44.617061 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:44.612713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:34373 (local address 127.0.0.1:39021)
0619 14:55:44.612817 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:44.612822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:44.612834 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:55:44.612860 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:44.612863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:44.612901 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:44.612977 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:44.613286 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:44.614011 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:44.615193 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:44.615465 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:44.615550 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:44.616461 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:44.616465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:44.616467 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:44.616477 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:44.616500 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:44.616512 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:44.616583 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:44.616933 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 14:55:44.616961 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:55:44.621090 8233 catalog_manager.cc:1347] Generated new cluster ID: 4bce22b0e885424f80197ff828dc7804
I20260619 14:55:44.621141 8233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:44.638527 8233 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:44.639488 8233 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:44.647648 8233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ee4fb6b9a7747dd94b6e1f56682e039: Generated new TSK 0
I20260619 14:55:44.648257 8233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:44.655357 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 8149
W20260619 14:55:44.659194 8134 connection.cc:535] client connection to 127.0.0.1:39021 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39021): Connection reset by peer (error 104)
2026-06-19T14:55:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1271 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:44 dist-test-slave-3qdh krb5kdc[8248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:44 dist-test-slave-3qdh krb5kdc[8248](info): set up 2 sockets
Jun 19 14:55:44 dist-test-slave-3qdh krb5kdc[8248](info): commencing operation
krb5kdc: starting...
W20260619 14:55:45.604406 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:45 dist-test-slave-3qdh krb5kdc[8248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:55:45.710705 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44365
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:45.804741 8261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:45.805068 8261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:45.805128 8261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:45.810932 8261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:45.811000 8261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:45.811088 8261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:45.811124 8261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:45.811146 8261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:45.816088 8261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:45.816867 8261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:45.817626 8261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:45.824689 8266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:45.824738 8269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:45.824746 8267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:45.824877 8261 server_base.cc:1034] running on GCE node
I20260619 14:55:45.825372 8261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:45.826236 8261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:45.827405 8261 hybrid_clock.cc:648] HybridClock initialized: now 1781880945827392 us; error 27 us; skew 500 ppm
Jun 19 14:55:45 dist-test-slave-3qdh krb5kdc[8248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880945, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:45.829872 8261 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:55:45.830611 8261 webserver.cc:460] Webserver started at http://127.0.0.1:41509/ using document root <none> and password file <none>
I20260619 14:55:45.831100 8261 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:45.831197 8261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:45.831383 8261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:45.833117 8261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "7ad60997424f4603869eb3f085b2f70b"
format_stamp: "Formatted at 2026-06-19 14:55:45 on dist-test-slave-3qdh"
server_key: "1892a2550fc2d4305c2575240418d282"
server_key_iv: "a9628fb4d1dc07b5afae1089d9c30cbf"
server_key_version: "encryptionkey@0"
I20260619 14:55:45.833606 8261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "7ad60997424f4603869eb3f085b2f70b"
format_stamp: "Formatted at 2026-06-19 14:55:45 on dist-test-slave-3qdh"
server_key: "1892a2550fc2d4305c2575240418d282"
server_key_iv: "a9628fb4d1dc07b5afae1089d9c30cbf"
server_key_version: "encryptionkey@0"
I20260619 14:55:45.836675 8261 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:55:45.838835 8276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:45.839916 8261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:45.840031 8261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "7ad60997424f4603869eb3f085b2f70b"
format_stamp: "Formatted at 2026-06-19 14:55:45 on dist-test-slave-3qdh"
server_key: "1892a2550fc2d4305c2575240418d282"
server_key_iv: "a9628fb4d1dc07b5afae1089d9c30cbf"
server_key_version: "encryptionkey@0"
I20260619 14:55:45.840121 8261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:45.844030 8261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:45.846698 8261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:45.846855 8261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:45.853902 8261 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44365
I20260619 14:55:45.853960 8328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44365 every 8 connection(s)
I20260619 14:55:45.854718 8261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:45.855073 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 8261
I20260619 14:55:45.855199 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:45.855515 19593 external_mini_cluster.cc:1428] Setting key 32b8887f25e8fe1a760f5f0e2e32f8a8
I20260619 14:55:45.857266 8329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:45.862006 8329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b: Bootstrap starting.
I20260619 14:55:45.864368 8329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:45.864698 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:45.857309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:42647 (local address 127.0.0.1:44365)
0619 14:55:45.857729 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:45.857749 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:45.860757 (+ 3008us) server_negotiation.cc:407] Connection header received
0619 14:55:45.861745 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:45.861768 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:45.862116 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:45.862442 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:45.863117 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:45.863122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:45.863137 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:45.863149 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:45.863206 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:45.863233 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:45.863366 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:45.863441 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 14:55:45.863727 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":118,"threads_started":1}
I20260619 14:55:45.865176 8329 log.cc:824] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:45.867193 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:45.867240 8329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b: No bootstrap required, opened a new log
I20260619 14:55:45.870424 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:45.868992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:55557 (local address 127.0.0.1:44365)
0619 14:55:45.869092 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:45.869095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:45.869319 (+ 224us) server_negotiation.cc:407] Connection header received
0619 14:55:45.869432 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:45.869434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:45.869457 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:45.869534 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:45.870103 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:45.870106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:45.870108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:45.870118 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:45.870139 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:45.870148 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:45.870232 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:45.870295 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 14:55:45.870327 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 14:55:45.871956 8329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad60997424f4603869eb3f085b2f70b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44365 } }
I20260619 14:55:45.872215 8329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:45.872269 8329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad60997424f4603869eb3f085b2f70b, State: Initialized, Role: FOLLOWER
I20260619 14:55:45.872735 8329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad60997424f4603869eb3f085b2f70b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44365 } }
I20260619 14:55:45.872891 8329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:45.872969 8329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:45.873116 8329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:45.874202 8329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad60997424f4603869eb3f085b2f70b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44365 } }
I20260619 14:55:45.874604 8329 leader_election.cc:302] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ad60997424f4603869eb3f085b2f70b; no voters:
I20260619 14:55:45.874955 8329 leader_election.cc:288] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:45.875046 8339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:45.875284 8339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b [term 1 LEADER]: Becoming Leader. State: Replica: 7ad60997424f4603869eb3f085b2f70b, State: Running, Role: LEADER
I20260619 14:55:45.875578 8339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad60997424f4603869eb3f085b2f70b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44365 } }
I20260619 14:55:45.876210 8329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:45.876776 8341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ad60997424f4603869eb3f085b2f70b. Latest consensus state: current_term: 1 leader_uuid: "7ad60997424f4603869eb3f085b2f70b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad60997424f4603869eb3f085b2f70b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44365 } } }
I20260619 14:55:45.876955 8341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:45.877055 8340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ad60997424f4603869eb3f085b2f70b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad60997424f4603869eb3f085b2f70b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44365 } } }
I20260619 14:55:45.877131 8340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:45.877401 8346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:45.880173 8346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:45.884690 8346 catalog_manager.cc:1347] Generated new cluster ID: a176dc14488f4ac59a65cf0839d04c55
I20260619 14:55:45.884744 8346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:45.913036 8346 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:45.913322 8346 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:45.932845 8346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ad60997424f4603869eb3f085b2f70b: Generated new TSK 0
I20260619 14:55:45.933573 8346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:45.947052 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 8261
2026-06-19T14:55:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:45 dist-test-slave-3qdh krb5kdc[8360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:45 dist-test-slave-3qdh krb5kdc[8360](info): set up 2 sockets
Jun 19 14:55:45 dist-test-slave-3qdh krb5kdc[8360](info): commencing operation
krb5kdc: starting...
W20260619 14:55:46.900068 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:46 dist-test-slave-3qdh krb5kdc[8360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:55:47.003154 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37223
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:47.100543 8373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:47.100762 8373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:47.100800 8373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:47.104578 8373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:47.104627 8373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:47.104646 8373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:47.104692 8373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:47.104728 8373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:47.109635 8373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37223
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:47.110536 8373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:47.111442 8373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:47.119046 8381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:47.119071 8379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:47.119120 8373 server_base.cc:1034] running on GCE node
W20260619 14:55:47.119076 8378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:47.119846 8373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:47.120826 8373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:47.122014 8373 hybrid_clock.cc:648] HybridClock initialized: now 1781880947122004 us; error 28 us; skew 500 ppm
Jun 19 14:55:47 dist-test-slave-3qdh krb5kdc[8360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:47.132509 8373 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:55:47.133296 8373 webserver.cc:460] Webserver started at http://127.0.0.1:46705/ using document root <none> and password file <none>
I20260619 14:55:47.133921 8373 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:47.134017 8373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:47.134241 8373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:47.136124 8373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "1df1d63e67044a6d8aabcc0d7b16316c"
format_stamp: "Formatted at 2026-06-19 14:55:47 on dist-test-slave-3qdh"
server_key: "12a72ef590f9f8bf8952a991e292d032"
server_key_iv: "4eab3d9d2efae6e54e8775f3f3d83e96"
server_key_version: "encryptionkey@0"
I20260619 14:55:47.136714 8373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "1df1d63e67044a6d8aabcc0d7b16316c"
format_stamp: "Formatted at 2026-06-19 14:55:47 on dist-test-slave-3qdh"
server_key: "12a72ef590f9f8bf8952a991e292d032"
server_key_iv: "4eab3d9d2efae6e54e8775f3f3d83e96"
server_key_version: "encryptionkey@0"
I20260619 14:55:47.140066 8373 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:55:47.142329 8388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:47.143435 8373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:47.143571 8373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "1df1d63e67044a6d8aabcc0d7b16316c"
format_stamp: "Formatted at 2026-06-19 14:55:47 on dist-test-slave-3qdh"
server_key: "12a72ef590f9f8bf8952a991e292d032"
server_key_iv: "4eab3d9d2efae6e54e8775f3f3d83e96"
server_key_version: "encryptionkey@0"
I20260619 14:55:47.143677 8373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:47.151965 8373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:47.154558 8373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:47.154763 8373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:47.161556 8373 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37223
I20260619 14:55:47.161583 8440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37223 every 8 connection(s)
I20260619 14:55:47.162343 8373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:47.164884 8441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:47.167728 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 8373
I20260619 14:55:47.167840 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:47.168131 19593 external_mini_cluster.cc:1428] Setting key 388d04dfbad3d295a37883bbc8b8fa18
I20260619 14:55:47.170357 8441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c: Bootstrap starting.
I20260619 14:55:47.172410 8441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:47.173105 8441 log.cc:824] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:47.174805 8441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c: No bootstrap required, opened a new log
Jun 19 14:55:47 dist-test-slave-3qdh krb5kdc[8360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:55:47.178831 8441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df1d63e67044a6d8aabcc0d7b16316c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37223 } }
I20260619 14:55:47.179085 8441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:47.179153 8441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1df1d63e67044a6d8aabcc0d7b16316c, State: Initialized, Role: FOLLOWER
I20260619 14:55:47.179579 8441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df1d63e67044a6d8aabcc0d7b16316c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37223 } }
I20260619 14:55:47.179689 8441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:47.179754 8441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:47.179860 8441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:47.180936 8441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df1d63e67044a6d8aabcc0d7b16316c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37223 } }
I20260619 14:55:47.181286 8441 leader_election.cc:302] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1df1d63e67044a6d8aabcc0d7b16316c; no voters:
I20260619 14:55:47.181599 8441 leader_election.cc:288] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:47.181710 8446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:47.181923 8446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c [term 1 LEADER]: Becoming Leader. State: Replica: 1df1d63e67044a6d8aabcc0d7b16316c, State: Running, Role: LEADER
I20260619 14:55:47.182236 8446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df1d63e67044a6d8aabcc0d7b16316c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37223 } }
I20260619 14:55:47.182423 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:47.169452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:36845 (local address 127.0.0.1:37223)
0619 14:55:47.169880 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:47.169894 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:47.169926 (+ 32us) server_negotiation.cc:407] Connection header received
0619 14:55:47.170596 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:47.170614 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:47.170934 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:47.171248 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:47.172045 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:47.173126 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:47.174184 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:47.174636 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:47.174801 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:47.177423 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:47.177432 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:47.177462 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:47.177494 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:47.179324 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:47.179880 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:47.179886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:47.179890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:47.180016 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:47.180361 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:47.180365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:47.180367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:47.180843 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:47.181079 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:47.181328 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 14:55:47.181524 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":135,"threads_started":1}
I20260619 14:55:47.182723 8441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:47.183720 8447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1df1d63e67044a6d8aabcc0d7b16316c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df1d63e67044a6d8aabcc0d7b16316c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37223 } } }
I20260619 14:55:47.183822 8447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:47.184216 8448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1df1d63e67044a6d8aabcc0d7b16316c. Latest consensus state: current_term: 1 leader_uuid: "1df1d63e67044a6d8aabcc0d7b16316c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1df1d63e67044a6d8aabcc0d7b16316c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37223 } } }
I20260619 14:55:47.184377 8448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:47.184672 8455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:47.185377 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:47.187791 8455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:47.192883 8455 catalog_manager.cc:1347] Generated new cluster ID: 3be8ac752d254a2aa061f8f6cfa97538
I20260619 14:55:47.192932 8455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:47.196455 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:47.188360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:37853 (local address 127.0.0.1:37223)
0619 14:55:47.188470 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:47.188473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:47.188791 (+ 318us) server_negotiation.cc:407] Connection header received
0619 14:55:47.188940 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:47.188945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:47.188990 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:47.189081 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:47.189644 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:47.190339 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:47.191301 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:47.191611 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:47.191723 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:47.193137 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:47.193141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:47.193143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:47.193176 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:47.194760 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:47.195285 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:47.195288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:47.195290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:47.195369 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:47.195659 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:47.195662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:47.195663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:47.195843 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:47.195974 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:47.196207 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 14:55:47.196323 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:55:47.216522 8455 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:47.217310 8455 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:47.224481 8455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1df1d63e67044a6d8aabcc0d7b16316c: Generated new TSK 0
I20260619 14:55:47.225063 8455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:47.229117 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 8373
W20260619 14:55:47.234123 8358 connection.cc:535] client connection to 127.0.0.1:37223 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37223): Connection reset by peer (error 104)
2026-06-19T14:55:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1282 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:47 dist-test-slave-3qdh krb5kdc[8472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:47 dist-test-slave-3qdh krb5kdc[8472](info): set up 2 sockets
Jun 19 14:55:47 dist-test-slave-3qdh krb5kdc[8472](info): commencing operation
krb5kdc: starting...
W20260619 14:55:48.204336 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:48 dist-test-slave-3qdh krb5kdc[8472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:55:48.311347 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35909
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:48.409492 8485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:48.409726 8485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:48.409791 8485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:48.413404 8485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:48.413483 8485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:48.413529 8485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:48.413563 8485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:48.413623 8485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:48.418270 8485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35909
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:48.419103 8485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:48.419919 8485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:48.427405 8491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:48.427413 8490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:48.427436 8493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:48.427443 8485 server_base.cc:1034] running on GCE node
I20260619 14:55:48.428164 8485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:48.429095 8485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:48.430277 8485 hybrid_clock.cc:648] HybridClock initialized: now 1781880948430267 us; error 33 us; skew 500 ppm
Jun 19 14:55:48 dist-test-slave-3qdh krb5kdc[8472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:48.432786 8485 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:55:48.433488 8485 webserver.cc:460] Webserver started at http://127.0.0.1:43701/ using document root <none> and password file <none>
I20260619 14:55:48.434041 8485 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:48.434135 8485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:48.434342 8485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:48.436168 8485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "5c979c58b4314bf7a2cd564f98418d02"
format_stamp: "Formatted at 2026-06-19 14:55:48 on dist-test-slave-3qdh"
server_key: "5f4c4c2c01d8f37f87c2cd80fca2bb37"
server_key_iv: "bba408b84bd655b4bd2ebead6bcb14d7"
server_key_version: "encryptionkey@0"
I20260619 14:55:48.436815 8485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "5c979c58b4314bf7a2cd564f98418d02"
format_stamp: "Formatted at 2026-06-19 14:55:48 on dist-test-slave-3qdh"
server_key: "5f4c4c2c01d8f37f87c2cd80fca2bb37"
server_key_iv: "bba408b84bd655b4bd2ebead6bcb14d7"
server_key_version: "encryptionkey@0"
I20260619 14:55:48.440042 8485 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:55:48.442222 8500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:48.443270 8485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 14:55:48.443398 8485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "5c979c58b4314bf7a2cd564f98418d02"
format_stamp: "Formatted at 2026-06-19 14:55:48 on dist-test-slave-3qdh"
server_key: "5f4c4c2c01d8f37f87c2cd80fca2bb37"
server_key_iv: "bba408b84bd655b4bd2ebead6bcb14d7"
server_key_version: "encryptionkey@0"
I20260619 14:55:48.443497 8485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:48.462224 8485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:48.464942 8485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:48.465123 8485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:48.471060 8485 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35909
I20260619 14:55:48.471098 8552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35909 every 8 connection(s)
I20260619 14:55:48.471817 8485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:48.474010 8553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:48.476248 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 8485
I20260619 14:55:48.476387 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:48.476655 19593 external_mini_cluster.cc:1428] Setting key 756666062bf2d955ade8e7aad688911d
I20260619 14:55:48.479022 8553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02: Bootstrap starting.
I20260619 14:55:48.480952 8553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:48.481594 8553 log.cc:824] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:48.483072 8553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02: No bootstrap required, opened a new log
I20260619 14:55:48.484347 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:48.477858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:59121 (local address 127.0.0.1:35909)
0619 14:55:48.478215 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:48.478227 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:48.478255 (+ 28us) server_negotiation.cc:407] Connection header received
0619 14:55:48.478762 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:48.478776 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:48.479072 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:48.479412 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:48.479752 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:48.480685 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:48.481578 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:48.482016 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:48.482177 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:48.482879 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:48.482883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:48.482893 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:48.483247 (+ 354us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:48.483294 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:48.483314 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:48.483434 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:48.483599 (+ 165us) server_negotiation.cc:299] Negotiation successful
0619 14:55:48.483713 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":111,"threads_started":1}
I20260619 14:55:48.486693 8553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c979c58b4314bf7a2cd564f98418d02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35909 } }
I20260619 14:55:48.486924 8553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:48.486981 8553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c979c58b4314bf7a2cd564f98418d02, State: Initialized, Role: FOLLOWER
I20260619 14:55:48.486991 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:48.487323 8553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c979c58b4314bf7a2cd564f98418d02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35909 } }
I20260619 14:55:48.487424 8553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:48.487498 8553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:48.487582 8553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:48.488426 8553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c979c58b4314bf7a2cd564f98418d02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35909 } }
I20260619 14:55:48.488754 8553 leader_election.cc:302] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c979c58b4314bf7a2cd564f98418d02; no voters:
I20260619 14:55:48.489046 8553 leader_election.cc:288] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:48.489207 8562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:48.489428 8562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02 [term 1 LEADER]: Becoming Leader. State: Replica: 5c979c58b4314bf7a2cd564f98418d02, State: Running, Role: LEADER
I20260619 14:55:48.489724 8562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c979c58b4314bf7a2cd564f98418d02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35909 } }
I20260619 14:55:48.490226 8553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:48.490677 8562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c979c58b4314bf7a2cd564f98418d02. Latest consensus state: current_term: 1 leader_uuid: "5c979c58b4314bf7a2cd564f98418d02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c979c58b4314bf7a2cd564f98418d02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35909 } } }
I20260619 14:55:48.490785 8562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:48.491158 8564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c979c58b4314bf7a2cd564f98418d02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c979c58b4314bf7a2cd564f98418d02" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35909 } } }
I20260619 14:55:48.491256 8564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:48.493399 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:48.489001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:58697 (local address 127.0.0.1:35909)
0619 14:55:48.489105 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:48.489109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:48.489282 (+ 173us) server_negotiation.cc:407] Connection header received
0619 14:55:48.489450 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:48.489453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:48.489492 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:48.489535 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:48.489912 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:48.490555 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:48.491702 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:48.491996 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:48.492070 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:48.492977 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:48.492980 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:48.492982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:48.492993 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:48.493018 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:48.493028 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:48.493110 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:48.493244 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 14:55:48.493271 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260619 14:55:48.494626 8578 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:55:48.494702 8578 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:55:48.494755 8569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:48.495549 8569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:48.499840 8569 catalog_manager.cc:1347] Generated new cluster ID: 42cfc44409014abb93d682eae7bc95e9
I20260619 14:55:48.499893 8569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:48.515708 8569 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:48.516547 8569 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:48.523592 8569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c979c58b4314bf7a2cd564f98418d02: Generated new TSK 0
I20260619 14:55:48.524124 8569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:48.529917 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 8485
W20260619 14:55:48.534687 8470 connection.cc:535] client connection to 127.0.0.1:35909 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35909): Connection reset by peer (error 104)
2026-06-19T14:55:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:48 dist-test-slave-3qdh krb5kdc[8584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:48 dist-test-slave-3qdh krb5kdc[8584](info): set up 2 sockets
Jun 19 14:55:48 dist-test-slave-3qdh krb5kdc[8584](info): commencing operation
krb5kdc: starting...
W20260619 14:55:49.520418 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.972s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:49 dist-test-slave-3qdh krb5kdc[8584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:55:49.632149 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40603
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:49.733127 8597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:49.733385 8597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:49.733450 8597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:49.737294 8597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:49.737375 8597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:49.737412 8597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:49.737468 8597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:49.737505 8597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:49.742328 8597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40603
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:49.743250 8597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:49.744196 8597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:49.751778 8602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:49.751780 8605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:49.751756 8603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:49.751963 8597 server_base.cc:1034] running on GCE node
I20260619 14:55:49.752681 8597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:49.753679 8597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:49.754865 8597 hybrid_clock.cc:648] HybridClock initialized: now 1781880949754847 us; error 28 us; skew 500 ppm
Jun 19 14:55:49 dist-test-slave-3qdh krb5kdc[8584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:49.757434 8597 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:55:49.758248 8597 webserver.cc:460] Webserver started at http://127.0.0.1:39159/ using document root <none> and password file <none>
I20260619 14:55:49.758790 8597 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:49.758875 8597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:49.759091 8597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:49.761003 8597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "22605c8475cb4495960705f1483ebb81"
format_stamp: "Formatted at 2026-06-19 14:55:49 on dist-test-slave-3qdh"
server_key: "b3912bbaf90cfde7820b6a5d5d0f61fb"
server_key_iv: "5e22be3fcc343a4174d7804d401caa0a"
server_key_version: "encryptionkey@0"
I20260619 14:55:49.761502 8597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "22605c8475cb4495960705f1483ebb81"
format_stamp: "Formatted at 2026-06-19 14:55:49 on dist-test-slave-3qdh"
server_key: "b3912bbaf90cfde7820b6a5d5d0f61fb"
server_key_iv: "5e22be3fcc343a4174d7804d401caa0a"
server_key_version: "encryptionkey@0"
I20260619 14:55:49.764887 8597 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:55:49.767115 8612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:49.768345 8597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:49.768481 8597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "22605c8475cb4495960705f1483ebb81"
format_stamp: "Formatted at 2026-06-19 14:55:49 on dist-test-slave-3qdh"
server_key: "b3912bbaf90cfde7820b6a5d5d0f61fb"
server_key_iv: "5e22be3fcc343a4174d7804d401caa0a"
server_key_version: "encryptionkey@0"
I20260619 14:55:49.768579 8597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:49.772756 8597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:49.775250 8597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:49.775431 8597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:49.781946 8597 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40603
I20260619 14:55:49.781960 8664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40603 every 8 connection(s)
I20260619 14:55:49.782718 8597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:49.785116 8665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:49.786700 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 8597
I20260619 14:55:49.786844 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:49.787092 19593 external_mini_cluster.cc:1428] Setting key 99bb0190d326d7cda821407777254bd1
I20260619 14:55:49.791000 8665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81: Bootstrap starting.
I20260619 14:55:49.791707 8668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:49.788253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:47719 (local address 127.0.0.1:40603)
0619 14:55:49.788656 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:49.788671 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:49.788701 (+ 30us) server_negotiation.cc:407] Connection header received
0619 14:55:49.789257 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:49.789273 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:49.789498 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:49.789733 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:49.790544 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:49.790548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:49.790559 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:49.790568 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:49.790613 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:49.790630 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:49.790732 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:49.790838 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 14:55:49.790992 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":126,"threads_started":1}
I20260619 14:55:49.793118 8665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:49.793820 8665 log.cc:824] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:49.794219 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:49.795914 8665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81: No bootstrap required, opened a new log
I20260619 14:55:49.797853 8668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:49.796392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.122:46545 (local address 127.0.0.1:40603)
0619 14:55:49.796517 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:49.796521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:49.796674 (+ 153us) server_negotiation.cc:407] Connection header received
0619 14:55:49.796763 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:49.796766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:49.796797 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:49.796877 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:49.797504 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:49.797507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:49.797510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:49.797522 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:49.797547 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:49.797559 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:49.797638 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:49.797708 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 14:55:49.797745 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:55:49.800016 8665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22605c8475cb4495960705f1483ebb81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40603 } }
I20260619 14:55:49.800206 8665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:49.800279 8665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22605c8475cb4495960705f1483ebb81, State: Initialized, Role: FOLLOWER
I20260619 14:55:49.800709 8665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22605c8475cb4495960705f1483ebb81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40603 } }
I20260619 14:55:49.800840 8665 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:49.800904 8665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:49.800990 8665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:49.801757 8665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22605c8475cb4495960705f1483ebb81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40603 } }
I20260619 14:55:49.802062 8665 leader_election.cc:302] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22605c8475cb4495960705f1483ebb81; no voters:
I20260619 14:55:49.802380 8665 leader_election.cc:288] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:49.802480 8675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:49.802680 8675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81 [term 1 LEADER]: Becoming Leader. State: Replica: 22605c8475cb4495960705f1483ebb81, State: Running, Role: LEADER
I20260619 14:55:49.802937 8675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22605c8475cb4495960705f1483ebb81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40603 } }
I20260619 14:55:49.803401 8665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:49.804471 8677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22605c8475cb4495960705f1483ebb81. Latest consensus state: current_term: 1 leader_uuid: "22605c8475cb4495960705f1483ebb81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22605c8475cb4495960705f1483ebb81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40603 } } }
I20260619 14:55:49.804522 8676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22605c8475cb4495960705f1483ebb81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22605c8475cb4495960705f1483ebb81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40603 } } }
I20260619 14:55:49.804592 8677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:49.804607 8676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:49.804942 8684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:49.807518 8684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:49.812286 8684 catalog_manager.cc:1347] Generated new cluster ID: 7d15c1ede8d349fcbe8a6dd68605c882
I20260619 14:55:49.812366 8684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:49.823968 8684 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:49.824249 8684 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:49.830476 8684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22605c8475cb4495960705f1483ebb81: Generated new TSK 0
I20260619 14:55:49.831123 8684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:49.838937 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 8597
2026-06-19T14:55:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:49 dist-test-slave-3qdh krb5kdc[8696](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:49 dist-test-slave-3qdh krb5kdc[8696](info): set up 2 sockets
Jun 19 14:55:49 dist-test-slave-3qdh krb5kdc[8696](info): commencing operation
krb5kdc: starting...
W20260619 14:55:50.848274 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:50 dist-test-slave-3qdh krb5kdc[8696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:55:50.957108 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:38281
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:42233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:38281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:51.058892 8709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:51.059141 8709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:51.059216 8709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:51.063046 8709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:51.063125 8709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:51.063179 8709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:51.063215 8709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:51.063248 8709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:51.068166 8709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:42233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:38281
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:38281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:51.069052 8709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:51.069953 8709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:51.077659 8717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:51.077826 8715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:51.077869 8714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:51.077868 8709 server_base.cc:1034] running on GCE node
I20260619 14:55:51.078467 8709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:51.079429 8709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:51.080590 8709 hybrid_clock.cc:648] HybridClock initialized: now 1781880951080572 us; error 28 us; skew 500 ppm
Jun 19 14:55:51 dist-test-slave-3qdh krb5kdc[8696](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880951, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:51.083118 8709 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:55:51.083897 8709 webserver.cc:460] Webserver started at http://127.19.34.126:39663/ using document root <none> and password file <none>
I20260619 14:55:51.084475 8709 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:51.084533 8709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:51.084764 8709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:51.086529 8709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "38e0c3affab84f51a49f5a29c3807874"
format_stamp: "Formatted at 2026-06-19 14:55:51 on dist-test-slave-3qdh"
server_key: "a24d3ade06a2fe3fe90a4330fe8808b0"
server_key_iv: "9633c459e90e6bfbcb5495626e678826"
server_key_version: "encryptionkey@0"
I20260619 14:55:51.087018 8709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "38e0c3affab84f51a49f5a29c3807874"
format_stamp: "Formatted at 2026-06-19 14:55:51 on dist-test-slave-3qdh"
server_key: "a24d3ade06a2fe3fe90a4330fe8808b0"
server_key_iv: "9633c459e90e6bfbcb5495626e678826"
server_key_version: "encryptionkey@0"
I20260619 14:55:51.090309 8709 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260619 14:55:51.092389 8724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:51.093446 8709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:55:51.093586 8709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "38e0c3affab84f51a49f5a29c3807874"
format_stamp: "Formatted at 2026-06-19 14:55:51 on dist-test-slave-3qdh"
server_key: "a24d3ade06a2fe3fe90a4330fe8808b0"
server_key_iv: "9633c459e90e6bfbcb5495626e678826"
server_key_version: "encryptionkey@0"
I20260619 14:55:51.093681 8709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:51.112735 8709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:51.115664 8709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:51.115883 8709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:51.122752 8709 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:38281
I20260619 14:55:51.122777 8776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:38281 every 8 connection(s)
I20260619 14:55:51.123582 8709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:51.126127 8777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:51.131103 8777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874: Bootstrap starting.
I20260619 14:55:51.131963 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 8709
I20260619 14:55:51.132071 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:51.132342 19593 external_mini_cluster.cc:1428] Setting key 886710f42c88d415c320691ad4a2229a
I20260619 14:55:51.133646 8777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:51.134497 8777 log.cc:824] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:51.136487 8777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874: No bootstrap required, opened a new log
I20260619 14:55:51.140153 8777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e0c3affab84f51a49f5a29c3807874" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38281 } }
I20260619 14:55:51.140393 8777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:51.140468 8777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e0c3affab84f51a49f5a29c3807874, State: Initialized, Role: FOLLOWER
Jun 19 14:55:51 dist-test-slave-3qdh krb5kdc[8696](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:55:51.140880 8777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e0c3affab84f51a49f5a29c3807874" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38281 } }
I20260619 14:55:51.141000 8777 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:51.141057 8777 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:51.141175 8777 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:51.142277 8777 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e0c3affab84f51a49f5a29c3807874" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38281 } }
I20260619 14:55:51.142675 8777 leader_election.cc:302] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38e0c3affab84f51a49f5a29c3807874; no voters:
I20260619 14:55:51.142966 8777 leader_election.cc:288] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:51.143049 8782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:51.143244 8782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874 [term 1 LEADER]: Becoming Leader. State: Replica: 38e0c3affab84f51a49f5a29c3807874, State: Running, Role: LEADER
I20260619 14:55:51.143502 8782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e0c3affab84f51a49f5a29c3807874" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38281 } }
I20260619 14:55:51.144220 8777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:51.145257 8783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38e0c3affab84f51a49f5a29c3807874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e0c3affab84f51a49f5a29c3807874" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38281 } } }
I20260619 14:55:51.145254 8784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38e0c3affab84f51a49f5a29c3807874. Latest consensus state: current_term: 1 leader_uuid: "38e0c3affab84f51a49f5a29c3807874" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38e0c3affab84f51a49f5a29c3807874" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 38281 } } }
I20260619 14:55:51.145403 8783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:51.145700 8784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:51.145879 8791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:51.147809 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:51.133350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51803 (local address 127.19.34.126:38281)
0619 14:55:51.133806 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:51.133826 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:51.133869 (+ 43us) server_negotiation.cc:407] Connection header received
0619 14:55:51.134706 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:51.134727 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:51.135083 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:51.135424 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:51.136039 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:51.137446 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:51.138242 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:51.138974 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:51.139167 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:51.141617 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:51.141622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:51.141636 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:51.141664 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:51.143512 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:51.144214 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:51.144220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:51.144226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:51.144404 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:51.144918 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:51.144922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:51.144923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:51.145463 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:51.146523 (+ 1060us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:51.146726 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 14:55:51.146946 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":117,"threads_started":1}
I20260619 14:55:51.148289 8791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:51.151041 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:51.156659 8791 catalog_manager.cc:1347] Generated new cluster ID: ae951ca1ab764c559106546d409b764b
I20260619 14:55:51.156714 8791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:51.160426 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:51.153430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45907 (local address 127.19.34.126:38281)
0619 14:55:51.153525 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:51.153529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:51.153544 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:55:51.153591 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:51.153594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:51.153633 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:51.153723 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:51.154466 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:51.154958 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:51.155823 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:51.156046 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:51.156149 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:51.157444 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:51.157448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:51.157450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:51.157471 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:51.158771 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:51.159209 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:51.159212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:51.159214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:51.159285 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:51.159583 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:51.159586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:51.159587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:51.159736 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:51.159877 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:51.160129 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 14:55:51.160223 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:55:51.176254 8791 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:51.177007 8791 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:51.191636 8791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38e0c3affab84f51a49f5a29c3807874: Generated new TSK 0
I20260619 14:55:51.192231 8791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:51.197834 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 8709
W20260619 14:55:51.202759 8694 connection.cc:535] client connection to 127.19.34.126:38281 recv error: Network error: failed to read from TLS socket (remote: 127.19.34.126:38281): Connection reset by peer (error 104)
2026-06-19T14:55:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:51 dist-test-slave-3qdh krb5kdc[8808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:51 dist-test-slave-3qdh krb5kdc[8808](info): set up 2 sockets
Jun 19 14:55:51 dist-test-slave-3qdh krb5kdc[8808](info): commencing operation
krb5kdc: starting...
W20260619 14:55:52.208600 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:52 dist-test-slave-3qdh krb5kdc[8808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:55:52.326692 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:42981
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:42981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:52.431260 8821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:52.431502 8821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:52.431545 8821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:52.435513 8821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:52.435567 8821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:52.435587 8821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:52.435603 8821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:52.435618 8821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:52.440855 8821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:42981
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:42981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:52.441871 8821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:52.442847 8821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:52.451056 8826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:52.451311 8821 server_base.cc:1034] running on GCE node
W20260619 14:55:52.451071 8829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:52.451067 8827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:52.452059 8821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:52.453154 8821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:52.454351 8821 hybrid_clock.cc:648] HybridClock initialized: now 1781880952454326 us; error 42 us; skew 500 ppm
Jun 19 14:55:52 dist-test-slave-3qdh krb5kdc[8808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880952, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:52.461920 8821 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:55:52.462785 8821 webserver.cc:460] Webserver started at http://127.19.34.126:34431/ using document root <none> and password file <none>
I20260619 14:55:52.463291 8821 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:52.463348 8821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:52.463512 8821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:52.465473 8821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "7b510a298b1048a7b8df1555b990444c"
format_stamp: "Formatted at 2026-06-19 14:55:52 on dist-test-slave-3qdh"
server_key: "8d85845cb18669fa0e33a63f73981098"
server_key_iv: "d72afa71dd5fb4b46abf2149ca3d0746"
server_key_version: "encryptionkey@0"
I20260619 14:55:52.466017 8821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "7b510a298b1048a7b8df1555b990444c"
format_stamp: "Formatted at 2026-06-19 14:55:52 on dist-test-slave-3qdh"
server_key: "8d85845cb18669fa0e33a63f73981098"
server_key_iv: "d72afa71dd5fb4b46abf2149ca3d0746"
server_key_version: "encryptionkey@0"
I20260619 14:55:52.469542 8821 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:55:52.471835 8836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:52.473019 8821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:52.473125 8821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "7b510a298b1048a7b8df1555b990444c"
format_stamp: "Formatted at 2026-06-19 14:55:52 on dist-test-slave-3qdh"
server_key: "8d85845cb18669fa0e33a63f73981098"
server_key_iv: "d72afa71dd5fb4b46abf2149ca3d0746"
server_key_version: "encryptionkey@0"
I20260619 14:55:52.473203 8821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:52.499969 8821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:52.502434 8821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:52.502559 8821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:52.509223 8821 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:42981
I20260619 14:55:52.509256 8888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:42981 every 8 connection(s)
I20260619 14:55:52.510043 8821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:52.512286 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 8821
I20260619 14:55:52.512451 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:52.512567 8889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:52.512799 19593 external_mini_cluster.cc:1428] Setting key a7afae769bac43d024198c1559b23ab2
I20260619 14:55:52.518376 8889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c: Bootstrap starting.
I20260619 14:55:52.520567 8889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:52.520776 8892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:52.514029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47021 (local address 127.19.34.126:42981)
0619 14:55:52.514426 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:52.514439 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:52.514469 (+ 30us) server_negotiation.cc:407] Connection header received
0619 14:55:52.515064 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:52.515079 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:52.515375 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:52.515708 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:52.516076 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:52.517049 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:52.517901 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:52.518303 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:52.518451 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:52.519280 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:52.519285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:52.519295 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:52.519604 (+ 309us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:52.519647 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:52.519679 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:52.519804 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:52.519955 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 14:55:52.520077 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":134,"threads_started":1}
I20260619 14:55:52.521271 8889 log.cc:824] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:52.523201 8889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c: No bootstrap required, opened a new log
I20260619 14:55:52.523300 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:52.526568 8889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b510a298b1048a7b8df1555b990444c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42981 } }
I20260619 14:55:52.526739 8889 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:52.526791 8889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b510a298b1048a7b8df1555b990444c, State: Initialized, Role: FOLLOWER
I20260619 14:55:52.527163 8889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b510a298b1048a7b8df1555b990444c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42981 } }
I20260619 14:55:52.527266 8889 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:52.527345 8889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:52.527451 8889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:52.528312 8889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b510a298b1048a7b8df1555b990444c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42981 } }
I20260619 14:55:52.528616 8889 leader_election.cc:302] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b510a298b1048a7b8df1555b990444c; no voters:
I20260619 14:55:52.528934 8889 leader_election.cc:288] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:52.529038 8899 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:52.529199 8892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:52.525180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50231 (local address 127.19.34.126:42981)
0619 14:55:52.525307 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:52.525312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:52.525526 (+ 214us) server_negotiation.cc:407] Connection header received
0619 14:55:52.525680 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:52.525682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:52.525718 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:52.525788 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:52.526031 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:52.526454 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:52.527442 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:52.527713 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:52.527857 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:52.528653 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:52.528656 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:52.528667 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:52.528679 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:52.528707 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:52.528717 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:52.528816 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:52.529027 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 14:55:52.529062 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:55:52.529304 8899 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c [term 1 LEADER]: Becoming Leader. State: Replica: 7b510a298b1048a7b8df1555b990444c, State: Running, Role: LEADER
I20260619 14:55:52.529606 8899 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b510a298b1048a7b8df1555b990444c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42981 } }
I20260619 14:55:52.530129 8889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:52.530647 8899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b510a298b1048a7b8df1555b990444c. Latest consensus state: current_term: 1 leader_uuid: "7b510a298b1048a7b8df1555b990444c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b510a298b1048a7b8df1555b990444c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42981 } } }
I20260619 14:55:52.530850 8899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:52.531090 8900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b510a298b1048a7b8df1555b990444c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b510a298b1048a7b8df1555b990444c" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42981 } } }
I20260619 14:55:52.531193 8900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:52.531324 8908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:52.533803 8908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:52.538764 8908 catalog_manager.cc:1347] Generated new cluster ID: 632b9a6cd067440c99c670cba1e02e11
I20260619 14:55:52.538818 8908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:52.560989 8908 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:52.562398 8908 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:52.567281 8908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b510a298b1048a7b8df1555b990444c: Generated new TSK 0
I20260619 14:55:52.567997 8908 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:52.575050 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 8821
W20260619 14:55:52.580389 8806 connection.cc:535] client connection to 127.19.34.126:42981 recv error: Network error: failed to read from TLS socket (remote: 127.19.34.126:42981): Connection reset by peer (error 104)
2026-06-19T14:55:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:52 dist-test-slave-3qdh krb5kdc[8920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:52 dist-test-slave-3qdh krb5kdc[8920](info): set up 2 sockets
Jun 19 14:55:52 dist-test-slave-3qdh krb5kdc[8920](info): commencing operation
krb5kdc: starting...
W20260619 14:55:53.576555 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.982s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:53 dist-test-slave-3qdh krb5kdc[8920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:55:53.686772 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:41011
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:45471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:41011
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:53.792404 8933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:53.792655 8933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:53.792694 8933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:53.796589 8933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:53.796641 8933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:53.796662 8933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:53.796718 8933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:53.796759 8933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:53.801594 8933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:45471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:41011
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:41011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:53.802466 8933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:53.803393 8933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:55:53.811539 8933 server_base.cc:1034] running on GCE node
W20260619 14:55:53.811477 8941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:53.811456 8939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:53.811478 8938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:53.812381 8933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:53.813390 8933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:53.814610 8933 hybrid_clock.cc:648] HybridClock initialized: now 1781880953814587 us; error 41 us; skew 500 ppm
Jun 19 14:55:53 dist-test-slave-3qdh krb5kdc[8920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880953, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:53.817436 8933 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:55:53.818310 8933 webserver.cc:460] Webserver started at http://127.19.34.126:34085/ using document root <none> and password file <none>
I20260619 14:55:53.818836 8933 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:53.818931 8933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:53.819167 8933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:53.821090 8933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "df18cc4c11a743c3befbca260fd24d17"
format_stamp: "Formatted at 2026-06-19 14:55:53 on dist-test-slave-3qdh"
server_key: "33fcfa7f09bb55848b75576aa32dcdbe"
server_key_iv: "13a097340725b9353cf5ebb8cbd580e7"
server_key_version: "encryptionkey@0"
I20260619 14:55:53.821686 8933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "df18cc4c11a743c3befbca260fd24d17"
format_stamp: "Formatted at 2026-06-19 14:55:53 on dist-test-slave-3qdh"
server_key: "33fcfa7f09bb55848b75576aa32dcdbe"
server_key_iv: "13a097340725b9353cf5ebb8cbd580e7"
server_key_version: "encryptionkey@0"
I20260619 14:55:53.825206 8933 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:55:53.827430 8948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:53.828584 8933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 14:55:53.828718 8933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "df18cc4c11a743c3befbca260fd24d17"
format_stamp: "Formatted at 2026-06-19 14:55:53 on dist-test-slave-3qdh"
server_key: "33fcfa7f09bb55848b75576aa32dcdbe"
server_key_iv: "13a097340725b9353cf5ebb8cbd580e7"
server_key_version: "encryptionkey@0"
I20260619 14:55:53.828828 8933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:53.833205 8933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:53.836411 8933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:53.836647 8933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:53.843735 8933 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:41011
I20260619 14:55:53.843745 9000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:41011 every 8 connection(s)
I20260619 14:55:53.844631 8933 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:53.847080 9001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:53.852042 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 8933
I20260619 14:55:53.852190 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:53.852244 9001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17: Bootstrap starting.
I20260619 14:55:53.852519 19593 external_mini_cluster.cc:1428] Setting key 19d6d05523917faea15f7d408907e794
I20260619 14:55:53.854657 9001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:53.855373 9001 log.cc:824] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:53.857482 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:53.853722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57215 (local address 127.19.34.126:41011)
0619 14:55:53.854175 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:53.854190 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:53.854225 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:55:53.854864 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:53.854880 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:53.855139 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:53.855399 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:53.856181 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:53.856186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:53.856202 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:53.856219 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:53.856282 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:53.856331 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:53.856446 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:53.856538 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 14:55:53.856669 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":127,"threads_started":1}
I20260619 14:55:53.857532 9001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17: No bootstrap required, opened a new log
I20260619 14:55:53.860076 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:53.861497 9001 raft_consensus.cc:348] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df18cc4c11a743c3befbca260fd24d17" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41011 } }
I20260619 14:55:53.861696 9001 raft_consensus.cc:374] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:53.861791 9001 raft_consensus.cc:729] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df18cc4c11a743c3befbca260fd24d17, State: Initialized, Role: FOLLOWER
I20260619 14:55:53.862223 9001 consensus_queue.cc:260] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df18cc4c11a743c3befbca260fd24d17" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41011 } }
I20260619 14:55:53.862396 9001 raft_consensus.cc:388] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:53.862473 9001 raft_consensus.cc:482] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:53.862584 9001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:53.863682 9004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:53.862051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36437 (local address 127.19.34.126:41011)
0619 14:55:53.862154 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:53.862165 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:53.862337 (+ 172us) server_negotiation.cc:407] Connection header received
0619 14:55:53.862460 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:53.862463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:53.862501 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:53.862594 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:53.863289 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:53.863293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:53.863295 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:53.863308 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:53.863334 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:53.863349 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:53.863468 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:53.863520 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 14:55:53.863554 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:55:53.863813 9001 raft_consensus.cc:504] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df18cc4c11a743c3befbca260fd24d17" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41011 } }
I20260619 14:55:53.864107 9001 leader_election.cc:302] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df18cc4c11a743c3befbca260fd24d17; no voters:
I20260619 14:55:53.864461 9001 leader_election.cc:288] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:53.864550 9011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:53.864759 9011 raft_consensus.cc:686] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17 [term 1 LEADER]: Becoming Leader. State: Replica: df18cc4c11a743c3befbca260fd24d17, State: Running, Role: LEADER
I20260619 14:55:53.865041 9011 consensus_queue.cc:237] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df18cc4c11a743c3befbca260fd24d17" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41011 } }
I20260619 14:55:53.865583 9001 sys_catalog.cc:564] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:53.866580 9013 sys_catalog.cc:455] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df18cc4c11a743c3befbca260fd24d17. Latest consensus state: current_term: 1 leader_uuid: "df18cc4c11a743c3befbca260fd24d17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df18cc4c11a743c3befbca260fd24d17" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41011 } } }
I20260619 14:55:53.866717 9013 sys_catalog.cc:458] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:53.867069 9020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:53.867049 9012 sys_catalog.cc:455] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df18cc4c11a743c3befbca260fd24d17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df18cc4c11a743c3befbca260fd24d17" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 41011 } } }
I20260619 14:55:53.867779 9012 sys_catalog.cc:458] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17 [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:53.870388 9020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:53.875237 9020 catalog_manager.cc:1347] Generated new cluster ID: ebd94e6abcfb4690a5b954df80d99617
I20260619 14:55:53.875306 9020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:53.889602 9020 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:53.889863 9020 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:53.899238 9020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df18cc4c11a743c3befbca260fd24d17: Generated new TSK 0
I20260619 14:55:53.899780 9020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:53.910014 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 8933
2026-06-19T14:55:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:53 dist-test-slave-3qdh krb5kdc[9032](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:53 dist-test-slave-3qdh krb5kdc[9032](info): set up 2 sockets
Jun 19 14:55:53 dist-test-slave-3qdh krb5kdc[9032](info): commencing operation
krb5kdc: starting...
W20260619 14:55:54.936312 19593 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:55 dist-test-slave-3qdh krb5kdc[9032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:55:55.047178 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:42051
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:42051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:55.147332 9045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:55.147555 9045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:55.147596 9045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:55.151472 9045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:55.151516 9045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:55.151533 9045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:55.151548 9045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:55.151561 9045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:55.156196 9045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:42051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:42051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:55.157092 9045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:55.157971 9045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:55.165607 9053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:55.165624 9051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:55.165617 9050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:55.165855 9045 server_base.cc:1034] running on GCE node
I20260619 14:55:55.166383 9045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:55.167312 9045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:55.168460 9045 hybrid_clock.cc:648] HybridClock initialized: now 1781880955168436 us; error 43 us; skew 500 ppm
Jun 19 14:55:55 dist-test-slave-3qdh krb5kdc[9032](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880955, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:55.171011 9045 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:55:55.171743 9045 webserver.cc:460] Webserver started at http://127.19.34.126:44651/ using document root <none> and password file <none>
I20260619 14:55:55.172236 9045 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:55.172291 9045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:55.172524 9045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:55.174350 9045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "1cf7dde36dd249dabec6390f9fbf16ab"
format_stamp: "Formatted at 2026-06-19 14:55:55 on dist-test-slave-3qdh"
server_key: "75bd168503069eeaaa6b1ef0b39570b8"
server_key_iv: "d4aa4b755aa51ab96528e6aa747b4cb1"
server_key_version: "encryptionkey@0"
I20260619 14:55:55.174876 9045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "1cf7dde36dd249dabec6390f9fbf16ab"
format_stamp: "Formatted at 2026-06-19 14:55:55 on dist-test-slave-3qdh"
server_key: "75bd168503069eeaaa6b1ef0b39570b8"
server_key_iv: "d4aa4b755aa51ab96528e6aa747b4cb1"
server_key_version: "encryptionkey@0"
I20260619 14:55:55.178145 9045 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:55:55.180286 9060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:55.181449 9045 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:55.181578 9045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "1cf7dde36dd249dabec6390f9fbf16ab"
format_stamp: "Formatted at 2026-06-19 14:55:55 on dist-test-slave-3qdh"
server_key: "75bd168503069eeaaa6b1ef0b39570b8"
server_key_iv: "d4aa4b755aa51ab96528e6aa747b4cb1"
server_key_version: "encryptionkey@0"
I20260619 14:55:55.181684 9045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:55.191507 9045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:55.194034 9045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:55.194221 9045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:55.200970 9045 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:42051
I20260619 14:55:55.201241 9112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:42051 every 8 connection(s)
I20260619 14:55:55.201812 9045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:55.204229 9113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:55.209051 9113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab: Bootstrap starting.
I20260619 14:55:55.211265 9113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:55.211963 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 9045
I20260619 14:55:55.211954 9113 log.cc:824] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:55.212102 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:55.212540 19593 external_mini_cluster.cc:1428] Setting key 5f973caf292cb4c0804134da99bf5a92
I20260619 14:55:55.213992 9113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab: No bootstrap required, opened a new log
I20260619 14:55:55.216719 9113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf7dde36dd249dabec6390f9fbf16ab" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42051 } }
I20260619 14:55:55.216926 9113 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:55.217005 9113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cf7dde36dd249dabec6390f9fbf16ab, State: Initialized, Role: FOLLOWER
I20260619 14:55:55.217455 9113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf7dde36dd249dabec6390f9fbf16ab" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42051 } }
I20260619 14:55:55.217593 9113 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:55.217677 9113 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:55.217775 9113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:55.218600 9113 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf7dde36dd249dabec6390f9fbf16ab" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42051 } }
I20260619 14:55:55.218920 9113 leader_election.cc:302] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cf7dde36dd249dabec6390f9fbf16ab; no voters:
I20260619 14:55:55.219226 9113 leader_election.cc:288] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:55.219329 9118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:55.219799 9118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab [term 1 LEADER]: Becoming Leader. State: Replica: 1cf7dde36dd249dabec6390f9fbf16ab, State: Running, Role: LEADER
I20260619 14:55:55.220126 9118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf7dde36dd249dabec6390f9fbf16ab" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42051 } }
I20260619 14:55:55.220682 9113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab [sys.catalog]: configured and running, proceeding with master startup.
Jun 19 14:55:55 dist-test-slave-3qdh krb5kdc[9032](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:55:55.221547 9120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cf7dde36dd249dabec6390f9fbf16ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf7dde36dd249dabec6390f9fbf16ab" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42051 } } }
I20260619 14:55:55.221688 9120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:55.222373 9119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cf7dde36dd249dabec6390f9fbf16ab. Latest consensus state: current_term: 1 leader_uuid: "1cf7dde36dd249dabec6390f9fbf16ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cf7dde36dd249dabec6390f9fbf16ab" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 42051 } } }
I20260619 14:55:55.222529 9119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:55.222852 9127 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:55.225565 9127 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:55.227312 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:55.213524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52363 (local address 127.19.34.126:42051)
0619 14:55:55.213880 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:55.213893 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:55.213922 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:55:55.214538 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:55.214554 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:55.214853 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:55.215150 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:55.215941 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:55.217378 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:55.218397 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:55.218843 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:55.219008 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:55.221978 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:55.221983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:55.221997 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:55.222025 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:55.224095 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:55.224613 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:55.224619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:55.224625 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:55.224754 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:55.225052 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:55.225055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:55.225057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:55.225524 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:55.225717 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:55.226002 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 14:55:55.226447 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":99,"threads_started":1}
I20260619 14:55:55.230418 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:55.230688 9127 catalog_manager.cc:1347] Generated new cluster ID: 907be7fe10174b809a0d4de6b4a4d06e
I20260619 14:55:55.230736 9127 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:55.237479 9127 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:55.238404 9127 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:55.239571 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:55.232329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49559 (local address 127.19.34.126:42051)
0619 14:55:55.232426 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:55.232430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:55.232652 (+ 222us) server_negotiation.cc:407] Connection header received
0619 14:55:55.232810 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:55.232812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:55.232857 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:55.232937 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:55.233535 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:55.234035 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:55.234800 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:55.235042 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:55.235176 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:55.236472 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:55.236476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:55.236478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:55.236501 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:55.237961 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:55.238333 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:55.238337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:55.238338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:55.238407 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:55.238754 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:55.238758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:55.238759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:55.238910 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:55.239052 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:55.239279 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 14:55:55.239399 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":41}
I20260619 14:55:55.246425 9127 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cf7dde36dd249dabec6390f9fbf16ab: Generated new TSK 0
I20260619 14:55:55.247040 9127 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:55.250270 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 9045
2026-06-19T14:55:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:55 dist-test-slave-3qdh krb5kdc[9144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:55 dist-test-slave-3qdh krb5kdc[9144](info): set up 2 sockets
Jun 19 14:55:55 dist-test-slave-3qdh krb5kdc[9144](info): commencing operation
krb5kdc: starting...
W20260619 14:55:56.244427 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:56 dist-test-slave-3qdh krb5kdc[9144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880956, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:55:56.355000 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:37979
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:37979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:56.455143 9157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:56.455350 9157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:56.455389 9157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:56.458935 9157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:56.458981 9157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:56.458997 9157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:56.459041 9157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:56.459065 9157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:56.463233 9157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:37979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:37979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:56.463996 9157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:56.464835 9157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:56.472397 9165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:56.472443 9163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:56.472445 9162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:56.472568 9157 server_base.cc:1034] running on GCE node
I20260619 14:55:56.473179 9157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:56.474105 9157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:56.475261 9157 hybrid_clock.cc:648] HybridClock initialized: now 1781880956475245 us; error 27 us; skew 500 ppm
Jun 19 14:55:56 dist-test-slave-3qdh krb5kdc[9144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880956, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:56.477810 9157 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:55:56.478611 9157 webserver.cc:460] Webserver started at http://127.19.34.126:46847/ using document root <none> and password file <none>
I20260619 14:55:56.479086 9157 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:56.479173 9157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:56.479379 9157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:56.481209 9157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "9499c8f189834a70b6eb453312db00ee"
format_stamp: "Formatted at 2026-06-19 14:55:56 on dist-test-slave-3qdh"
server_key: "7639074de2fa3b43cf2054e3d27a4fca"
server_key_iv: "dd4b36f8c33322841566d91e7ab6e617"
server_key_version: "encryptionkey@0"
I20260619 14:55:56.481765 9157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "9499c8f189834a70b6eb453312db00ee"
format_stamp: "Formatted at 2026-06-19 14:55:56 on dist-test-slave-3qdh"
server_key: "7639074de2fa3b43cf2054e3d27a4fca"
server_key_iv: "dd4b36f8c33322841566d91e7ab6e617"
server_key_version: "encryptionkey@0"
I20260619 14:55:56.485003 9157 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:55:56.487152 9172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:56.488183 9157 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:56.488346 9157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "9499c8f189834a70b6eb453312db00ee"
format_stamp: "Formatted at 2026-06-19 14:55:56 on dist-test-slave-3qdh"
server_key: "7639074de2fa3b43cf2054e3d27a4fca"
server_key_iv: "dd4b36f8c33322841566d91e7ab6e617"
server_key_version: "encryptionkey@0"
I20260619 14:55:56.488443 9157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:56.507123 9157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:56.509898 9157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:56.510071 9157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:56.516554 9157 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:37979
I20260619 14:55:56.516579 9224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:37979 every 8 connection(s)
I20260619 14:55:56.517365 9157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:56.519783 9225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:56.520062 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 9157
I20260619 14:55:56.520172 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:56.520457 19593 external_mini_cluster.cc:1428] Setting key 5c132d67c8d01169e50a7ec9f85065e0
I20260619 14:55:56.525374 9225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee: Bootstrap starting.
I20260619 14:55:56.527334 9225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:56.528074 9225 log.cc:824] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:56.528374 9228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:56.521711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54087 (local address 127.19.34.126:37979)
0619 14:55:56.522109 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:56.522122 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:56.522151 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:55:56.522736 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:56.522750 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:56.523026 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:56.523302 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:56.523622 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:56.524491 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:56.525388 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:56.525878 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:56.526056 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:56.526862 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:56.526867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:56.526877 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:56.527212 (+ 335us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:56.527272 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:56.527304 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:56.527399 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:56.527501 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 14:55:56.527673 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":117,"threads_started":1}
I20260619 14:55:56.529924 9225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee: No bootstrap required, opened a new log
I20260619 14:55:56.530910 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:56.534296 9225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9499c8f189834a70b6eb453312db00ee" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 37979 } }
I20260619 14:55:56.534530 9225 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:56.534590 9225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9499c8f189834a70b6eb453312db00ee, State: Initialized, Role: FOLLOWER
I20260619 14:55:56.535022 9225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9499c8f189834a70b6eb453312db00ee" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 37979 } }
I20260619 14:55:56.535171 9225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:56.535233 9225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:56.535321 9225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:56.536369 9225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9499c8f189834a70b6eb453312db00ee" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 37979 } }
I20260619 14:55:56.536728 9225 leader_election.cc:302] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9499c8f189834a70b6eb453312db00ee; no voters:
I20260619 14:55:56.537029 9228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:56.533109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39919 (local address 127.19.34.126:37979)
0619 14:55:56.533230 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:56.533235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:56.533373 (+ 138us) server_negotiation.cc:407] Connection header received
0619 14:55:56.533470 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:56.533472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:56.533504 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:56.533577 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:56.533852 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:56.534335 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:56.535466 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:56.535693 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:56.535771 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:56.536572 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:56.536576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:56.536579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:56.536597 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:56.536626 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:56.536637 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:56.536691 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:56.536817 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 14:55:56.536884 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:55:56.537046 9225 leader_election.cc:288] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:56.537338 9235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:56.537503 9235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee [term 1 LEADER]: Becoming Leader. State: Replica: 9499c8f189834a70b6eb453312db00ee, State: Running, Role: LEADER
I20260619 14:55:56.537774 9235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9499c8f189834a70b6eb453312db00ee" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 37979 } }
I20260619 14:55:56.538213 9225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:56.538764 9236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9499c8f189834a70b6eb453312db00ee. Latest consensus state: current_term: 1 leader_uuid: "9499c8f189834a70b6eb453312db00ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9499c8f189834a70b6eb453312db00ee" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 37979 } } }
I20260619 14:55:56.538904 9236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:56.539142 9237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9499c8f189834a70b6eb453312db00ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9499c8f189834a70b6eb453312db00ee" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 37979 } } }
I20260619 14:55:56.539227 9237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:56.539319 9243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:56.541850 9243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:56.546931 9243 catalog_manager.cc:1347] Generated new cluster ID: 04c4f6d839994fd3b64714d305ed13df
I20260619 14:55:56.546998 9243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:56.554288 9243 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:56.555027 9243 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:56.558604 9243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9499c8f189834a70b6eb453312db00ee: Generated new TSK 0
I20260619 14:55:56.559226 9243 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:56.570362 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 9157
2026-06-19T14:55:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:56 dist-test-slave-3qdh krb5kdc[9256](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:56 dist-test-slave-3qdh krb5kdc[9256](info): set up 2 sockets
Jun 19 14:55:56 dist-test-slave-3qdh krb5kdc[9256](info): commencing operation
krb5kdc: starting...
W20260619 14:55:57.528182 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:57 dist-test-slave-3qdh krb5kdc[9256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:55:57.635475 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:33363
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:42757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:33363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:57.728915 9269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:57.729099 9269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:57.729132 9269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:57.732409 9269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:57.732445 9269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:57.732460 9269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:57.732474 9269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:57.732486 9269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:57.736833 9269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:42757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:33363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:33363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:57.737506 9269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:57.738233 9269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:57.745057 9274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:57.745045 9275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:57.745107 9269 server_base.cc:1034] running on GCE node
W20260619 14:55:57.745036 9277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:57.745714 9269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:57.746724 9269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:57.747920 9269 hybrid_clock.cc:648] HybridClock initialized: now 1781880957747911 us; error 36 us; skew 500 ppm
Jun 19 14:55:57 dist-test-slave-3qdh krb5kdc[9256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880957, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:57.755033 9269 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:55:57.755807 9269 webserver.cc:460] Webserver started at http://127.19.34.126:33741/ using document root <none> and password file <none>
I20260619 14:55:57.756389 9269 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:57.756446 9269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:57.756644 9269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:57.758311 9269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "eb4f3b8fea7247f090b8e07803ea946d"
format_stamp: "Formatted at 2026-06-19 14:55:57 on dist-test-slave-3qdh"
server_key: "05288abfce616b12bc797b793b4aaa41"
server_key_iv: "595e646a03b2bf716e7f4f2a697e9bf8"
server_key_version: "encryptionkey@0"
I20260619 14:55:57.758769 9269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "eb4f3b8fea7247f090b8e07803ea946d"
format_stamp: "Formatted at 2026-06-19 14:55:57 on dist-test-slave-3qdh"
server_key: "05288abfce616b12bc797b793b4aaa41"
server_key_iv: "595e646a03b2bf716e7f4f2a697e9bf8"
server_key_version: "encryptionkey@0"
I20260619 14:55:57.761835 9269 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:55:57.763844 9284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:57.765064 9269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:57.765208 9269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "eb4f3b8fea7247f090b8e07803ea946d"
format_stamp: "Formatted at 2026-06-19 14:55:57 on dist-test-slave-3qdh"
server_key: "05288abfce616b12bc797b793b4aaa41"
server_key_iv: "595e646a03b2bf716e7f4f2a697e9bf8"
server_key_version: "encryptionkey@0"
I20260619 14:55:57.765301 9269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:57.769423 9269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:57.772164 9269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:57.772356 9269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:57.778306 9269 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:33363
I20260619 14:55:57.778378 9336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:33363 every 8 connection(s)
I20260619 14:55:57.779070 9269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:57.779551 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 9269
I20260619 14:55:57.779685 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:57.779929 19593 external_mini_cluster.cc:1428] Setting key 2f02a095e44b4138965351531160806b
I20260619 14:55:57.781754 9337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:57.784245 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:57.781077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43077 (local address 127.19.34.126:33363)
0619 14:55:57.781437 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:57.781451 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:57.781480 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:55:57.782065 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:57.782085 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:57.782331 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:57.782565 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:57.783243 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:57.783247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:57.783257 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:57.783264 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:57.783303 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:57.783317 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:57.783408 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:57.783500 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 14:55:57.783630 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":119,"threads_started":1}
I20260619 14:55:57.786780 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:57.787305 9337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d: Bootstrap starting.
I20260619 14:55:57.789364 9337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:57.789880 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:57.788564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56431 (local address 127.19.34.126:33363)
0619 14:55:57.788653 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:57.788657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:57.788795 (+ 138us) server_negotiation.cc:407] Connection header received
0619 14:55:57.788900 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:57.788902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:57.788923 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:57.788976 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:57.789547 (+ 571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:57.789551 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:57.789555 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:55:57.789569 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:57.789587 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:55:57.789602 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:57.789664 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:57.789752 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 14:55:57.789776 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:55:57.790148 9337 log.cc:824] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:57.791640 9337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d: No bootstrap required, opened a new log
I20260619 14:55:57.794402 9337 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb4f3b8fea7247f090b8e07803ea946d" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33363 } }
I20260619 14:55:57.794586 9337 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:55:57.794636 9337 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb4f3b8fea7247f090b8e07803ea946d, State: Initialized, Role: FOLLOWER
I20260619 14:55:57.795040 9337 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb4f3b8fea7247f090b8e07803ea946d" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33363 } }
I20260619 14:55:57.795183 9337 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:55:57.795246 9337 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:55:57.795334 9337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:57.796108 9337 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb4f3b8fea7247f090b8e07803ea946d" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33363 } }
I20260619 14:55:57.796463 9337 leader_election.cc:302] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb4f3b8fea7247f090b8e07803ea946d; no voters:
I20260619 14:55:57.796726 9337 leader_election.cc:288] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:55:57.796834 9347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:57.797041 9347 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d [term 1 LEADER]: Becoming Leader. State: Replica: eb4f3b8fea7247f090b8e07803ea946d, State: Running, Role: LEADER
I20260619 14:55:57.797338 9347 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb4f3b8fea7247f090b8e07803ea946d" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33363 } }
I20260619 14:55:57.797684 9337 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:57.798741 9348 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb4f3b8fea7247f090b8e07803ea946d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb4f3b8fea7247f090b8e07803ea946d" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33363 } } }
I20260619 14:55:57.798705 9349 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb4f3b8fea7247f090b8e07803ea946d. Latest consensus state: current_term: 1 leader_uuid: "eb4f3b8fea7247f090b8e07803ea946d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb4f3b8fea7247f090b8e07803ea946d" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33363 } } }
I20260619 14:55:57.798870 9349 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:57.799190 9358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:57.799117 9348 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:57.801489 9358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:57.805563 9358 catalog_manager.cc:1347] Generated new cluster ID: c7a96cd97a6440a9ba44b78e861562d4
I20260619 14:55:57.805624 9358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:57.814164 9358 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:57.814396 9358 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:57.819656 9358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb4f3b8fea7247f090b8e07803ea946d: Generated new TSK 0
I20260619 14:55:57.820192 9358 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:55:57.824815 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 9269
2026-06-19T14:55:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1253 ms)
[----------] 18 tests from AuthTokenIssuingTest (23447 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:55:57 dist-test-slave-3qdh krb5kdc[9368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:55:57 dist-test-slave-3qdh krb5kdc[9368](info): set up 2 sockets
Jun 19 14:55:57 dist-test-slave-3qdh krb5kdc[9368](info): commencing operation
krb5kdc: starting...
W20260619 14:55:58.768561 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:55:58 dist-test-slave-3qdh krb5kdc[9368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:55:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:55:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:55:58.876049 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:33233
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:33233,127.19.34.125:34579,127.19.34.124:46409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:55:58.974856 9381 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:55:58.975181 9381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:58.975235 9381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:58.975307 9381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:58.979620 9381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:58.979693 9381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:58.979728 9381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:58.979763 9381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:58.979794 9381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:58.984690 9381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:33233,127.19.34.125:34579,127.19.34.124:46409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:33233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:58.985682 9381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:58.986629 9381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:58.994091 9389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:58.994104 9386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:58.994179 9381 server_base.cc:1034] running on GCE node
W20260619 14:55:58.994096 9387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:58.994783 9381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:58.995704 9381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:58.996853 9381 hybrid_clock.cc:648] HybridClock initialized: now 1781880958996838 us; error 28 us; skew 500 ppm
Jun 19 14:55:58 dist-test-slave-3qdh krb5kdc[9368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880958, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:58.999064 9381 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:55:58.999759 9381 webserver.cc:460] Webserver started at http://127.19.34.126:40077/ using document root <none> and password file <none>
I20260619 14:55:59.000228 9381 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:59.000305 9381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:59.000516 9381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:59.002197 9381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "16239ee1b31c4b03a58943320c79032e"
format_stamp: "Formatted at 2026-06-19 14:55:58 on dist-test-slave-3qdh"
server_key: "83d24d5ef81f46ebec3a1471fed14e0a"
server_key_iv: "02ff0b1a7fce1d7ab6d3a0857a3b5338"
server_key_version: "encryptionkey@0"
I20260619 14:55:59.002732 9381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "16239ee1b31c4b03a58943320c79032e"
format_stamp: "Formatted at 2026-06-19 14:55:58 on dist-test-slave-3qdh"
server_key: "83d24d5ef81f46ebec3a1471fed14e0a"
server_key_iv: "02ff0b1a7fce1d7ab6d3a0857a3b5338"
server_key_version: "encryptionkey@0"
I20260619 14:55:59.006089 9381 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 14:55:59.008204 9396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:59.009236 9381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:59.009361 9381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "16239ee1b31c4b03a58943320c79032e"
format_stamp: "Formatted at 2026-06-19 14:55:58 on dist-test-slave-3qdh"
server_key: "83d24d5ef81f46ebec3a1471fed14e0a"
server_key_iv: "02ff0b1a7fce1d7ab6d3a0857a3b5338"
server_key_version: "encryptionkey@0"
I20260619 14:55:59.009454 9381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:59.029501 9381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:59.031975 9381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:59.032151 9381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:59.038568 9381 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:33233
I20260619 14:55:59.038843 9448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:33233 every 8 connection(s)
I20260619 14:55:59.039398 9381 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:55:59.040807 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 9381
I20260619 14:55:59.040931 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:55:59.041190 19593 external_mini_cluster.cc:1428] Setting key a9f86774d2356cc1c6103e5bd4fb6420
I20260619 14:55:59.042295 9449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:59.045850 9449 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33233 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.34.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.125@KRBTEST.COM" created.
Jun 19 14:55:59 dist-test-slave-3qdh krb5kdc[9368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880958, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:55:59.059995 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.047808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:33233 (local address 127.0.0.1:37582)
0619 14:55:59.048288 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.048370 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:59.049072 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:59.049650 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:59.049654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:59.049973 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:59.050199 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:59.050211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.051549 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.051556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:59.052535 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:59.052544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.053293 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.053299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:59.053448 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.053940 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:59.053962 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:59.055483 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:59.057293 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:59.057301 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:59.057309 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:59.057501 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:59.057777 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:59.057781 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:59.057783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:59.057880 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:59.058404 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:59.058409 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:59.058529 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 14:55:59.058706 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 14:55:59.059105 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":117,"thread_start_us":114,"threads_started":1}
I20260619 14:55:59.059995 9453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.047808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37582 (local address 127.19.34.126:33233)
0619 14:55:59.048269 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.048284 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.048461 (+ 177us) server_negotiation.cc:407] Connection header received
0619 14:55:59.049336 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.049340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.049489 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.049627 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.050332 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.051415 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.052638 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.053150 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.053254 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.055622 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:59.055629 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:59.055645 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:59.055664 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:59.057182 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.057598 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.057604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.057606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.057695 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.057966 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.057968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.057970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.058305 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:59.058426 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:59.058761 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 14:55:59.059024 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":344,"spinlock_wait_cycles":36608,"thread_start_us":114,"threads_started":1}
I20260619 14:55:59.062036 9449 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:55:59.062651 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.062432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:34579 (local address 127.0.0.1:52592)
0619 14:55:59.062522 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.062605 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.125:34579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 14:55:59.062808 9399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.34.125:34579: connect: Connection refused (error 111)
I20260619 14:55:59.063504 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.063307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:34579 (local address 127.0.0.1:52594)
0619 14:55:59.063388 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.063424 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.125:34579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 14:55:59.063777 9449 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.125:34579: Network error: Client connection negotiation failed: client connection to 127.19.34.125:34579: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.125.keytab.
Entry for principal kudu/127.19.34.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.125.keytab.
I20260619 14:55:59.064770 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.125:34579
--webserver_interface=127.19.34.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.125.keytab
--principal=kudu/127.19.34.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:33233,127.19.34.125:34579,127.19.34.124:46409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:55:59.113041 9449 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } attempt: 1
I20260619 14:55:59.113829 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.113545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:34579 (local address 127.0.0.1:52606)
0619 14:55:59.113684 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.113749 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.125:34579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 14:55:59.114514 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.114309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:34579 (local address 127.0.0.1:52608)
0619 14:55:59.114411 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.114460 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.125:34579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 14:55:59.114650 9449 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.125:34579: Network error: Client connection negotiation failed: client connection to 127.19.34.125:34579: connect: Connection refused (error 111)
W20260619 14:55:59.159337 9456 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:55:59.159593 9456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:59.159664 9456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:59.159736 9456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:59.163241 9456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:59.163311 9456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:59.163347 9456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:59.163398 9456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:59.163434 9456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:59.168000 9456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:33233,127.19.34.125:34579,127.19.34.124:46409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.125.keytab
--principal=kudu/127.19.34.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.125:34579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.34.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:59.169059 9456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:59.169955 9456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:55:59.177223 9462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:59.177232 9461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:59.177281 9456 server_base.cc:1034] running on GCE node
W20260619 14:55:59.177237 9464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:59.177935 9456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:59.178809 9456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:59.179999 9456 hybrid_clock.cc:648] HybridClock initialized: now 1781880959179972 us; error 52 us; skew 500 ppm
Jun 19 14:55:59 dist-test-slave-3qdh krb5kdc[9368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880959, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:59.182747 9456 init.cc:377] Logged in from keytab as kudu/127.19.34.125@KRBTEST.COM (short username kudu)
I20260619 14:55:59.182818 9449 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } attempt: 2
I20260619 14:55:59.183522 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.183273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:34579 (local address 127.0.0.1:52622)
0619 14:55:59.183406 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.183454 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.125:34579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 14:55:59.183733 9456 webserver.cc:460] Webserver started at http://127.19.34.125:45253/ using document root <none> and password file <none>
I20260619 14:55:59.184155 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.183983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:34579 (local address 127.0.0.1:52632)
0619 14:55:59.184073 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.184111 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.125:34579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 14:55:59.184331 9449 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.125:34579: Network error: Client connection negotiation failed: client connection to 127.19.34.125:34579: connect: Connection refused (error 111)
I20260619 14:55:59.184373 9456 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:59.184469 9456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:59.184654 9456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:59.186475 9456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-1/data/instance:
uuid: "519442b09558472fa92037ef77a7b6f2"
format_stamp: "Formatted at 2026-06-19 14:55:59 on dist-test-slave-3qdh"
server_key: "392a79ff44002c037a7193c3b1838778"
server_key_iv: "6ed68f05e1c8ced49abfa4fbd98d1d9f"
server_key_version: "encryptionkey@0"
I20260619 14:55:59.186988 9456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-1/wal/instance:
uuid: "519442b09558472fa92037ef77a7b6f2"
format_stamp: "Formatted at 2026-06-19 14:55:59 on dist-test-slave-3qdh"
server_key: "392a79ff44002c037a7193c3b1838778"
server_key_iv: "6ed68f05e1c8ced49abfa4fbd98d1d9f"
server_key_version: "encryptionkey@0"
I20260619 14:55:59.190232 9456 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:55:59.192431 9471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:59.193440 9456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:55:59.193567 9456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-1/wal
uuid: "519442b09558472fa92037ef77a7b6f2"
format_stamp: "Formatted at 2026-06-19 14:55:59 on dist-test-slave-3qdh"
server_key: "392a79ff44002c037a7193c3b1838778"
server_key_iv: "6ed68f05e1c8ced49abfa4fbd98d1d9f"
server_key_version: "encryptionkey@0"
I20260619 14:55:59.193662 9456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:59.209935 9456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:59.212709 9456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:59.212875 9456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:59.218958 9456 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.125:34579
I20260619 14:55:59.219246 9523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.125:34579 every 8 connection(s)
I20260619 14:55:59.219810 9456 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-1/data/info.pb
I20260619 14:55:59.222091 9524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:59.225101 9524 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33233 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:55:59.229408 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 9456
I20260619 14:55:59.229521 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-1/wal/instance
I20260619 14:55:59.229794 19593 external_mini_cluster.cc:1428] Setting key 130053d56e2a0629505bb9e99ba9ad52
Jun 19 14:55:59 dist-test-slave-3qdh krb5kdc[9368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880959, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.125@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.34.124@KRBTEST.COM; defaulting to no policy
I20260619 14:55:59.237704 9453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.226805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37594 (local address 127.19.34.126:33233)
0619 14:55:59.226944 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.226950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.227483 (+ 533us) server_negotiation.cc:407] Connection header received
0619 14:55:59.228392 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.228395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.228451 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.228559 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.229706 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.230187 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.231576 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.231829 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.231931 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.234340 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:59.234343 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:59.234345 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:59.234364 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:59.235739 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.236386 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.236389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.236391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.236464 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.236795 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.236797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.236798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.236985 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:59.237051 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:59.237500 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 14:55:59.237584 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:55:59.238356 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.227013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:33233 (local address 127.0.0.1:37594)
0619 14:55:59.227361 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.227394 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:59.228179 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:59.228711 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:59.228717 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:59.228997 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:59.229554 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:59.229574 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.230308 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.230313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:59.231460 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:59.231469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.231958 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.231961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:59.232103 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.232595 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:59.232609 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:59.234186 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:59.235854 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:59.235859 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:59.235873 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:59.236249 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:59.236561 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:59.236564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:59.236566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:59.236710 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:59.237079 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:59.237085 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:59.237308 (+ 223us) client_negotiation.cc:769] Sending connection context
0619 14:55:59.237433 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 14:55:59.237620 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":92,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.124@KRBTEST.COM" created.
I20260619 14:55:59.239557 9524 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:55:59 dist-test-slave-3qdh krb5kdc[9368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880959, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.125@KRBTEST.COM for kudu/127.19.34.125@KRBTEST.COM
I20260619 14:55:59.248239 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.239956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:34579 (local address 127.0.0.1:52642)
0619 14:55:59.240110 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.240143 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:59.240205 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:59.240584 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:59.240590 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:59.240750 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:59.240827 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:59.240832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.241998 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.242003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:59.242838 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:59.242851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.243347 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.243352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:59.243471 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.243859 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:59.243879 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:59.245104 (+ 1225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:59.246867 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:59.246872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:59.246875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:59.247039 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:59.247305 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:59.247307 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:59.247309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:59.247384 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:59.247775 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:59.247777 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:59.247860 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 14:55:59.247969 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 14:55:59.248076 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 14:55:59.248268 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.240019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52642 (local address 127.19.34.125:34579)
0619 14:55:59.240224 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.240232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.240250 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:55:59.240286 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.240289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.240468 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.240575 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.240925 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.241889 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.242942 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.243231 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.243325 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.245239 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:59.245247 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:59.245261 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:59.245285 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:59.246770 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.247133 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.247137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.247139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.247210 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.247477 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.247480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.247482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.247682 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:59.247805 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:59.248039 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 14:55:59.248144 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260619 14:55:59.249940 9524 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:55:59.250597 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.250359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:46409 (local address 127.0.0.1:55430)
0619 14:55:59.250454 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.250514 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 14:55:59.250802 9474 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.34.124:46409: connect: Connection refused (error 111)
I20260619 14:55:59.251717 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.251479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:46409 (local address 127.0.0.1:55434)
0619 14:55:59.251569 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.251638 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 14:55:59.251868 9524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.124:46409: Network error: Client connection negotiation failed: client connection to 127.19.34.124:46409: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.124.keytab.
Entry for principal kudu/127.19.34.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.124.keytab.
I20260619 14:55:59.253343 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.124:46409
--webserver_interface=127.19.34.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.124.keytab
--principal=kudu/127.19.34.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:35727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:33233,127.19.34.125:34579,127.19.34.124:46409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:55:59.275501 9449 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } attempt: 3
Jun 19 14:55:59 dist-test-slave-3qdh krb5kdc[9368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880958, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for kudu/127.19.34.125@KRBTEST.COM
I20260619 14:55:59.284548 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.276037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:34579 (local address 127.0.0.1:52646)
0619 14:55:59.276197 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.276213 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:59.276340 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:59.276658 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:59.276661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:59.276832 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:59.276939 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:59.276944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.277983 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.277986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:59.278990 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:59.279000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.279533 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.279535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:59.279609 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.280091 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:59.280102 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:59.281352 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:59.283171 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:59.283173 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:59.283175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:59.283417 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:59.283691 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:59.283693 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:59.283694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:59.283764 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:59.284122 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:59.284125 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:59.284203 (+ 78us) client_negotiation.cc:769] Sending connection context
0619 14:55:59.284279 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:55:59.284429 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 14:55:59.284647 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.276116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52646 (local address 127.19.34.125:34579)
0619 14:55:59.276263 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.276266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.276281 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:55:59.276472 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.276475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.276522 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.276603 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.277058 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.277854 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.279108 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.279431 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.279528 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.281497 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:59.281501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:59.281503 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:59.281526 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:59.283065 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.283510 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.283514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.283515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.283596 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.283845 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.283847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.283849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.284036 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:59.284131 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:59.284373 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 14:55:59.284487 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 14:55:59.285835 9449 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:55:59.286506 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.286274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:46409 (local address 127.0.0.1:55446)
0619 14:55:59.286396 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.286445 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 14:55:59.287180 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.286958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:46409 (local address 127.0.0.1:55450)
0619 14:55:59.287084 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.287127 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 14:55:59.287294 9449 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.124:46409: Network error: Client connection negotiation failed: client connection to 127.19.34.124:46409: connect: Connection refused (error 111)
I20260619 14:55:59.301100 9524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } attempt: 1
I20260619 14:55:59.301736 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.301497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:46409 (local address 127.0.0.1:55454)
0619 14:55:59.301616 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.301665 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 14:55:59.302393 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.302198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:46409 (local address 127.0.0.1:55464)
0619 14:55:59.302300 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.302348 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 14:55:59.302511 9524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.124:46409: Network error: Client connection negotiation failed: client connection to 127.19.34.124:46409: connect: Connection refused (error 111)
I20260619 14:55:59.318436 9449 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } attempt: 1
I20260619 14:55:59.319067 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.318836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:46409 (local address 127.0.0.1:55470)
0619 14:55:59.318948 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.319009 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 14:55:59.319734 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.319511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:46409 (local address 127.0.0.1:55486)
0619 14:55:59.319630 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.319679 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 14:55:59.319844 9449 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.124:46409: Network error: Client connection negotiation failed: client connection to 127.19.34.124:46409: connect: Connection refused (error 111)
W20260619 14:55:59.358753 9531 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:55:59.359019 9531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:55:59.359086 9531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:55:59.359167 9531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:55:59.363050 9531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:55:59.363124 9531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:55:59.363168 9531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:55:59.363236 9531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:55:59.363274 9531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:55:59.368100 9531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:35727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:33233,127.19.34.125:34579,127.19.34.124:46409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.124.keytab
--principal=kudu/127.19.34.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.124:46409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.34.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:55:59.369019 9531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:55:59.369890 9531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:55:59.370664 9524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } attempt: 2
I20260619 14:55:59.371351 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.371110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:46409 (local address 127.0.0.1:55490)
0619 14:55:59.371239 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.371290 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 14:55:59.371999 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.371806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:46409 (local address 127.0.0.1:55498)
0619 14:55:59.371902 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.371949 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 14:55:59.372112 9524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.124:46409: Network error: Client connection negotiation failed: client connection to 127.19.34.124:46409: connect: Connection refused (error 111)
W20260619 14:55:59.377712 9536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:59.377780 9537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:55:59.377722 9539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:55:59.377872 9531 server_base.cc:1034] running on GCE node
I20260619 14:55:59.378448 9531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:55:59.379304 9531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:55:59.380479 9531 hybrid_clock.cc:648] HybridClock initialized: now 1781880959380468 us; error 20 us; skew 500 ppm
Jun 19 14:55:59 dist-test-slave-3qdh krb5kdc[9368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880959, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:55:59.382822 9531 init.cc:377] Logged in from keytab as kudu/127.19.34.124@KRBTEST.COM (short username kudu)
I20260619 14:55:59.383562 9531 webserver.cc:460] Webserver started at http://127.19.34.124:44397/ using document root <none> and password file <none>
I20260619 14:55:59.384060 9531 fs_manager.cc:359] Metadata directory not provided
I20260619 14:55:59.384145 9531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:55:59.384421 9531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:55:59.386214 9531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-2/data/instance:
uuid: "08db7533e1114fc280efe6f3f3dd065d"
format_stamp: "Formatted at 2026-06-19 14:55:59 on dist-test-slave-3qdh"
server_key: "89e226f8c401f314a35100175b0c7fce"
server_key_iv: "cf2fc79465510442358ce2f664de3cff"
server_key_version: "encryptionkey@0"
I20260619 14:55:59.386694 9531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-2/wal/instance:
uuid: "08db7533e1114fc280efe6f3f3dd065d"
format_stamp: "Formatted at 2026-06-19 14:55:59 on dist-test-slave-3qdh"
server_key: "89e226f8c401f314a35100175b0c7fce"
server_key_iv: "cf2fc79465510442358ce2f664de3cff"
server_key_version: "encryptionkey@0"
I20260619 14:55:59.389958 9531 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:55:59.392024 9546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:55:59.393112 9531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:55:59.393250 9531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-2/wal
uuid: "08db7533e1114fc280efe6f3f3dd065d"
format_stamp: "Formatted at 2026-06-19 14:55:59 on dist-test-slave-3qdh"
server_key: "89e226f8c401f314a35100175b0c7fce"
server_key_iv: "cf2fc79465510442358ce2f664de3cff"
server_key_version: "encryptionkey@0"
I20260619 14:55:59.393345 9531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:55:59.394997 9449 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } attempt: 2
I20260619 14:55:59.395581 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.395350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:46409 (local address 127.0.0.1:55502)
0619 14:55:59.395466 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.395523 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 14:55:59.396231 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.396007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:46409 (local address 127.0.0.1:55518)
0619 14:55:59.396103 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.396163 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:46409: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 14:55:59.396345 9449 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.124:46409: Network error: Client connection negotiation failed: client connection to 127.19.34.124:46409: connect: Connection refused (error 111)
I20260619 14:55:59.407960 9531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:55:59.410415 9531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:55:59.410578 9531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:55:59.416538 9531 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.124:46409
I20260619 14:55:59.416775 9598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.124:46409 every 8 connection(s)
I20260619 14:55:59.417318 9531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-2/data/info.pb
I20260619 14:55:59.418305 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 9531
I20260619 14:55:59.418422 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880697016847-19593-0/minicluster-data/master-2/wal/instance
I20260619 14:55:59.418692 19593 external_mini_cluster.cc:1428] Setting key a3c80cd2ee2bd93e897b2a3d712655e4
I20260619 14:55:59.420086 9599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:55:59.423789 9599 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33233 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:55:59 dist-test-slave-3qdh krb5kdc[9368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.124@KRBTEST.COM
Jun 19 14:55:59 dist-test-slave-3qdh krb5kdc[9368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880959, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.124@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:55:59.431538 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.419801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.19.34.124:46409)
0619 14:55:59.420152 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.420203 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.420247 (+ 44us) server_negotiation.cc:407] Connection header received
0619 14:55:59.420877 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.420909 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.421198 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.421426 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.422247 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.423055 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.424038 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.424526 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.424628 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.426985 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:59.426990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:59.427000 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:59.427024 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:59.428898 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.429355 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.429359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.429363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.429470 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.429757 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.429759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.429760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.430043 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:59.430228 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:59.430468 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 14:55:59.430801 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":124,"threads_started":1}
I20260619 14:55:59.433598 9453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.425475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37598 (local address 127.19.34.126:33233)
0619 14:55:59.425625 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.425629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.425983 (+ 354us) server_negotiation.cc:407] Connection header received
0619 14:55:59.426107 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.426110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.426149 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.426243 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.426818 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.427464 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.428274 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.428605 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.428750 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.430589 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:59.430593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:59.430595 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:59.430622 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:59.431978 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.432417 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.432423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.432424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.432506 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.432761 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.432765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.432766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.432983 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:59.433059 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:59.433368 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 14:55:59.433455 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:55:59.433619 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.425433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:33233 (local address 127.0.0.1:37598)
0619 14:55:59.425844 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.425890 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:59.426010 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:59.426285 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:59.426289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:59.426564 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:59.426713 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:59.426719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.427575 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.427578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:59.428153 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:59.428161 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.428749 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.428751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:59.428827 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.429255 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:59.429268 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:59.430470 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:59.432084 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:59.432090 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:59.432093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:59.432294 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:59.432599 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:59.432602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:59.432604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:59.432680 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:59.433113 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:59.433117 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:59.433202 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 14:55:59.433320 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 14:55:59.433457 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":145,"threads_started":1}
I20260619 14:55:59.434718 9599 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:55:59 dist-test-slave-3qdh krb5kdc[9368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
Jun 19 14:55:59 dist-test-slave-3qdh krb5kdc[9368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880959, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.124@KRBTEST.COM for kudu/127.19.34.125@KRBTEST.COM
I20260619 14:55:59.441742 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.433396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37610 (local address 127.19.34.126:33233)
0619 14:55:59.433665 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.433673 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.433685 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:55:59.433772 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.433774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.433810 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.433888 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.434398 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.434897 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.436126 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.436422 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.436494 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.438883 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:59.438886 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:59.438887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:59.438916 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:59.440349 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.440710 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.440715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.440716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.440790 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.441022 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.441025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.441026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.441176 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:59.441336 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:59.441464 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 14:55:59.441579 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":94,"threads_started":1}
I20260619 14:55:59.441987 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.435157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:34579 (local address 127.0.0.1:52656)
0619 14:55:59.435245 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.435257 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:59.435326 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:59.435533 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:59.435535 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:59.435655 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:59.435730 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:59.435735 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.436593 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.436596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:59.437189 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:59.437198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.437817 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.437821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:59.437928 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.438261 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:59.438274 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:59.439265 (+ 991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:59.440792 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:59.440796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:59.440798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:59.440955 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:59.441197 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:59.441201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:59.441203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:59.441274 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:59.441594 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:59.441605 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:59.441675 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 14:55:59.441769 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 14:55:59.441869 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260619 14:55:59.442049 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.435262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.19.34.125:34579)
0619 14:55:59.435367 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.435371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.435386 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:55:59.435428 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.435430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.435464 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.435521 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.435840 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.436481 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.437302 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.437660 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.437778 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.439381 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:59.439384 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:59.439386 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:59.439408 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:59.440698 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.441038 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.441042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.441043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.441119 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.441349 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.441353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.441355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.441526 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:59.441603 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:59.441822 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 14:55:59.441922 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 14:55:59.442647 9599 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:55:59 dist-test-slave-3qdh krb5kdc[9368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880959, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.124@KRBTEST.COM for kudu/127.19.34.124@KRBTEST.COM
Jun 19 14:55:59 dist-test-slave-3qdh krb5kdc[9368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.125@KRBTEST.COM
I20260619 14:55:59.449954 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.443159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:46409 (local address 127.0.0.1:55542)
0619 14:55:59.443240 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.443252 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:59.443310 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:59.443492 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:59.443494 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:59.443626 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:59.443681 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:59.443685 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.444673 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.444675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:59.445156 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:59.445164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.445661 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.445665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:59.445768 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.446090 (+ 322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:59.446129 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:59.447213 (+ 1084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:59.448815 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:59.448819 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:59.448822 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:59.449066 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:59.449305 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:59.449309 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:59.449312 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:59.449386 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:59.449650 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:59.449652 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:59.449711 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 14:55:59.449779 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 14:55:59.449848 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260619 14:55:59.450117 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.443219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55542 (local address 127.19.34.124:46409)
0619 14:55:59.443306 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.443309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.443320 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:55:59.443382 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.443386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.443413 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.443475 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.443777 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.444567 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.445263 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.445531 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.445638 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.447326 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:59.447329 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:59.447332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:59.447354 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:59.448723 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.449167 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.449169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.449170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.449229 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.449462 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.449463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.449464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.449582 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:59.449641 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:59.449857 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 14:55:59.449951 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 14:55:59.451154 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.442975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52666 (local address 127.19.34.125:34579)
0619 14:55:59.443108 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.443112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.443124 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:55:59.443172 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.443175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.443212 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.443272 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.443818 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.444230 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.444905 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.445167 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.445268 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.447867 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:59.447870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:59.447872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:59.447892 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:59.449390 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.449967 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.449972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.449976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.450144 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.450410 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.450413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.450414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.450594 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:59.450811 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:59.450924 (+ 113us) server_negotiation.cc:299] Negotiation successful
0619 14:55:59.451023 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:55:59.452780 9599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08db7533e1114fc280efe6f3f3dd065d: Bootstrap starting.
I20260619 14:55:59.454066 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:55:59.454900 9599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08db7533e1114fc280efe6f3f3dd065d: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:59.455588 9599 log.cc:824] T 00000000000000000000000000000000 P 08db7533e1114fc280efe6f3f3dd065d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:59.455166 9423 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "16239ee1b31c4b03a58943320c79032e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37610
I20260619 14:55:59.457255 9599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08db7533e1114fc280efe6f3f3dd065d: No bootstrap required, opened a new log
I20260619 14:55:59.459668 9599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08db7533e1114fc280efe6f3f3dd065d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16239ee1b31c4b03a58943320c79032e" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33233 } } peers { permanent_uuid: "519442b09558472fa92037ef77a7b6f2" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } } peers { permanent_uuid: "08db7533e1114fc280efe6f3f3dd065d" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } }
I20260619 14:55:59.459879 9599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08db7533e1114fc280efe6f3f3dd065d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08db7533e1114fc280efe6f3f3dd065d, State: Initialized, Role: FOLLOWER
I20260619 14:55:59.460157 9599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08db7533e1114fc280efe6f3f3dd065d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16239ee1b31c4b03a58943320c79032e" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33233 } } peers { permanent_uuid: "519442b09558472fa92037ef77a7b6f2" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } } peers { permanent_uuid: "08db7533e1114fc280efe6f3f3dd065d" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } }
I20260619 14:55:59.460611 9606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08db7533e1114fc280efe6f3f3dd065d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16239ee1b31c4b03a58943320c79032e" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33233 } } peers { permanent_uuid: "519442b09558472fa92037ef77a7b6f2" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } } peers { permanent_uuid: "08db7533e1114fc280efe6f3f3dd065d" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } } }
I20260619 14:55:59.460743 9606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08db7533e1114fc280efe6f3f3dd065d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:55:59.461380 9599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08db7533e1114fc280efe6f3f3dd065d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:59.463232 9524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } attempt: 3
W20260619 14:55:59.465445 9617 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08db7533e1114fc280efe6f3f3dd065d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:55:59.465523 9617 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:55:59.464324 9531 master_runner.cc:186] Error getting master registration for 127.19.34.126:33233: OK, instance_id { permanent_uuid: "16239ee1b31c4b03a58943320c79032e" instance_seqno: 1781880959036820 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:55:59.467082 9531 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 19 14:55:59 dist-test-slave-3qdh krb5kdc[9368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880959, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.125@KRBTEST.COM for kudu/127.19.34.124@KRBTEST.COM
I20260619 14:55:59.471544 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.463636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55556 (local address 127.19.34.124:46409)
0619 14:55:59.463727 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.463730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.463823 (+ 93us) server_negotiation.cc:407] Connection header received
0619 14:55:59.463929 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.463932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.463970 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.464029 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.464502 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.465168 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.466345 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.466623 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.466715 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.468841 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:59.468845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:59.468847 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:59.468867 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:59.470118 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.470509 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.470513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.470514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.470586 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.470835 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.470839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.470841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.470988 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:59.471082 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:59.471337 (+ 255us) server_negotiation.cc:299] Negotiation successful
0619 14:55:59.471428 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:55:59.471544 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.463559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:46409 (local address 127.0.0.1:55556)
0619 14:55:59.463762 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.463774 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:59.463848 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:59.464124 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:59.464127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:59.464288 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:59.464392 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:59.464399 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.465332 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.465355 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:59.466227 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:59.466238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.466717 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.466735 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:59.466808 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.467308 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:59.467323 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:59.468722 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:59.470241 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:59.470243 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:59.470245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:59.470419 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:59.470675 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:59.470678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:59.470679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:59.470755 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:59.471075 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:59.471078 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:59.471182 (+ 104us) client_negotiation.cc:769] Sending connection context
0619 14:55:59.471293 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 14:55:59.471405 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260619 14:55:59.473845 9524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519442b09558472fa92037ef77a7b6f2: Bootstrap starting.
I20260619 14:55:59.475708 9524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 519442b09558472fa92037ef77a7b6f2: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:59.476383 9524 log.cc:824] T 00000000000000000000000000000000 P 519442b09558472fa92037ef77a7b6f2: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:59.477782 9524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519442b09558472fa92037ef77a7b6f2: No bootstrap required, opened a new log
I20260619 14:55:59.480154 9524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 519442b09558472fa92037ef77a7b6f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16239ee1b31c4b03a58943320c79032e" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33233 } } peers { permanent_uuid: "519442b09558472fa92037ef77a7b6f2" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } } peers { permanent_uuid: "08db7533e1114fc280efe6f3f3dd065d" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } }
I20260619 14:55:59.480355 9524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 519442b09558472fa92037ef77a7b6f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519442b09558472fa92037ef77a7b6f2, State: Initialized, Role: FOLLOWER
I20260619 14:55:59.480588 9524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 519442b09558472fa92037ef77a7b6f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16239ee1b31c4b03a58943320c79032e" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33233 } } peers { permanent_uuid: "519442b09558472fa92037ef77a7b6f2" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } } peers { permanent_uuid: "08db7533e1114fc280efe6f3f3dd065d" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } }
I20260619 14:55:59.480930 9620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519442b09558472fa92037ef77a7b6f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16239ee1b31c4b03a58943320c79032e" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33233 } } peers { permanent_uuid: "519442b09558472fa92037ef77a7b6f2" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } } peers { permanent_uuid: "08db7533e1114fc280efe6f3f3dd065d" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } } }
I20260619 14:55:59.481052 9620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519442b09558472fa92037ef77a7b6f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:55:59.481776 9524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 519442b09558472fa92037ef77a7b6f2 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:55:59.486513 9631 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 519442b09558472fa92037ef77a7b6f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:55:59.486620 9631 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:55:59.484771 9456 master_runner.cc:186] Error getting master registration for 127.19.34.126:33233: OK, instance_id { permanent_uuid: "16239ee1b31c4b03a58943320c79032e" instance_seqno: 1781880959036820 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:55:59.487576 9456 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:55:59.495502 9449 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } attempt: 3
Jun 19 14:55:59 dist-test-slave-3qdh krb5kdc[9368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880958, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for kudu/127.19.34.124@KRBTEST.COM
I20260619 14:55:59.503436 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.495938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:46409 (local address 127.0.0.1:55564)
0619 14:55:59.496131 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 14:55:59.496146 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:55:59.496265 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:55:59.496539 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:55:59.496542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:55:59.496719 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:55:59.496794 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:59.496799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.497633 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.497636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:59.498145 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:55:59.498152 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.498617 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.498620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:55:59.498714 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.499152 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:55:59.499169 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:55:59.500464 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:55:59.502155 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:59.502169 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:59.502171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:59.502343 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:59.502584 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:55:59.502586 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:55:59.502588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:55:59.502654 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:55:59.503000 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:55:59.503003 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:55:59.503086 (+ 83us) client_negotiation.cc:769] Sending connection context
0619 14:55:59.503180 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 14:55:59.503289 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 14:55:59.503436 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.496011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55564 (local address 127.19.34.124:46409)
0619 14:55:59.496129 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.496133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.496233 (+ 100us) server_negotiation.cc:407] Connection header received
0619 14:55:59.496391 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.496396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.496443 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.496507 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.496901 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.497541 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.498252 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.498535 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.498612 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.500583 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:59.500586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:59.500588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:59.500609 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:59.502062 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.502429 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.502433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.502436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.502508 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.502748 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.502751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.502752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.502921 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:59.503000 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:59.503226 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 14:55:59.503318 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:55:59.506553 9449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e: Bootstrap starting.
I20260619 14:55:59.509095 9449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e: Neither blocks nor log segments found. Creating new log.
I20260619 14:55:59.509837 9449 log.cc:824] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e: Log is configured to *not* fsync() on all Append() calls
I20260619 14:55:59.511577 9449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e: No bootstrap required, opened a new log
I20260619 14:55:59.514240 9449 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16239ee1b31c4b03a58943320c79032e" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33233 } } peers { permanent_uuid: "519442b09558472fa92037ef77a7b6f2" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } } peers { permanent_uuid: "08db7533e1114fc280efe6f3f3dd065d" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } }
I20260619 14:55:59.514417 9449 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16239ee1b31c4b03a58943320c79032e, State: Initialized, Role: FOLLOWER
I20260619 14:55:59.514720 9449 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16239ee1b31c4b03a58943320c79032e" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33233 } } peers { permanent_uuid: "519442b09558472fa92037ef77a7b6f2" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } } peers { permanent_uuid: "08db7533e1114fc280efe6f3f3dd065d" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } }
I20260619 14:55:59.515228 9634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16239ee1b31c4b03a58943320c79032e" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33233 } } peers { permanent_uuid: "519442b09558472fa92037ef77a7b6f2" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } } peers { permanent_uuid: "08db7533e1114fc280efe6f3f3dd065d" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } } }
I20260619 14:55:59.515453 9634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:55:59.515960 9449 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:55:59.516044 9423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 14:55:59.516129 9423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:59.517721 9423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16239ee1b31c4b03a58943320c79032e" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33233 } } peers { permanent_uuid: "519442b09558472fa92037ef77a7b6f2" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } } peers { permanent_uuid: "08db7533e1114fc280efe6f3f3dd065d" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } }
I20260619 14:55:59.518535 9423 leader_election.cc:288] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [CANDIDATE]: Term 1 election: Requested vote from peers 519442b09558472fa92037ef77a7b6f2 (127.19.34.125:34579), 08db7533e1114fc280efe6f3f3dd065d (127.19.34.124:46409)
I20260619 14:55:59.520116 9498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16239ee1b31c4b03a58943320c79032e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "519442b09558472fa92037ef77a7b6f2"
I20260619 14:55:59.520360 9498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 519442b09558472fa92037ef77a7b6f2 [term 0 FOLLOWER]: Advancing to term 1
W20260619 14:55:59.521323 9645 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:55:59.521377 9645 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:55:59.521595 9498 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 519442b09558472fa92037ef77a7b6f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16239ee1b31c4b03a58943320c79032e in term 1.
I20260619 14:55:59.521958 9573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16239ee1b31c4b03a58943320c79032e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "08db7533e1114fc280efe6f3f3dd065d"
I20260619 14:55:59.522140 9573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08db7533e1114fc280efe6f3f3dd065d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:55:59.523350 9573 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 08db7533e1114fc280efe6f3f3dd065d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16239ee1b31c4b03a58943320c79032e in term 1.
I20260619 14:55:59.525344 9399 leader_election.cc:302] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16239ee1b31c4b03a58943320c79032e, 519442b09558472fa92037ef77a7b6f2; no voters:
I20260619 14:55:59.530434 9381 master_runner.cc:263] No leader master found from master 16239ee1b31c4b03a58943320c79032e
I20260619 14:55:59.530508 9381 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:55:59.530668 9634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:55:59.530892 9634 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [term 1 LEADER]: Becoming Leader. State: Replica: 16239ee1b31c4b03a58943320c79032e, State: Running, Role: LEADER
I20260619 14:55:59.531235 9634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16239ee1b31c4b03a58943320c79032e" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33233 } } peers { permanent_uuid: "519442b09558472fa92037ef77a7b6f2" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } } peers { permanent_uuid: "08db7533e1114fc280efe6f3f3dd065d" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } }
I20260619 14:55:59.532889 9634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16239ee1b31c4b03a58943320c79032e. Latest consensus state: current_term: 1 leader_uuid: "16239ee1b31c4b03a58943320c79032e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16239ee1b31c4b03a58943320c79032e" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33233 } } peers { permanent_uuid: "519442b09558472fa92037ef77a7b6f2" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } } peers { permanent_uuid: "08db7533e1114fc280efe6f3f3dd065d" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } } }
I20260619 14:55:59.533030 9634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [sys.catalog]: This master's current role is: LEADER
W20260619 14:55:59.533738 9402 tablet.cc:2369] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:55:59.536532 9654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:55:59.537461 9654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:55:59.549464 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.533523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52672 (local address 127.19.34.125:34579)
0619 14:55:59.533618 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.533622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.536571 (+ 2949us) server_negotiation.cc:407] Connection header received
0619 14:55:59.536735 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.536738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.536776 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.536818 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.537677 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.538436 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.544383 (+ 5947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.544697 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.544760 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.546341 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:59.546345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:59.546347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:59.546370 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:59.547686 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.548172 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.548176 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.548178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.548251 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.548552 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.548555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.548557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.548742 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:59.548833 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:59.549223 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 14:55:59.549330 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:55:59.551774 9498 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 519442b09558472fa92037ef77a7b6f2 [term 1 FOLLOWER]: Refusing update from remote peer 16239ee1b31c4b03a58943320c79032e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:55:59.552495 9573 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08db7533e1114fc280efe6f3f3dd065d [term 1 FOLLOWER]: Refusing update from remote peer 16239ee1b31c4b03a58943320c79032e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:55:59.553176 9634 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [LEADER]: Connected to new peer: Peer: permanent_uuid: "08db7533e1114fc280efe6f3f3dd065d" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:55:59.553906 9634 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [LEADER]: Connected to new peer: Peer: permanent_uuid: "519442b09558472fa92037ef77a7b6f2" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:55:59.557230 9620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519442b09558472fa92037ef77a7b6f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16239ee1b31c4b03a58943320c79032e. Latest consensus state: current_term: 1 leader_uuid: "16239ee1b31c4b03a58943320c79032e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16239ee1b31c4b03a58943320c79032e" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33233 } } peers { permanent_uuid: "519442b09558472fa92037ef77a7b6f2" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } } peers { permanent_uuid: "08db7533e1114fc280efe6f3f3dd065d" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } } }
I20260619 14:55:59.557324 9620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519442b09558472fa92037ef77a7b6f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:55:59.559886 9606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08db7533e1114fc280efe6f3f3dd065d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16239ee1b31c4b03a58943320c79032e. Latest consensus state: current_term: 1 leader_uuid: "16239ee1b31c4b03a58943320c79032e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16239ee1b31c4b03a58943320c79032e" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33233 } } peers { permanent_uuid: "519442b09558472fa92037ef77a7b6f2" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } } peers { permanent_uuid: "08db7533e1114fc280efe6f3f3dd065d" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } } }
I20260619 14:55:59.559980 9606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08db7533e1114fc280efe6f3f3dd065d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:55:59.561484 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.529020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37620 (local address 127.19.34.126:33233)
0619 14:55:59.529136 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.529140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.535879 (+ 6739us) server_negotiation.cc:407] Connection header received
0619 14:55:59.536052 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.536055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.536097 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.536142 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.538552 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.539249 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.541659 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.541965 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.542028 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.545975 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:59.545979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:59.545982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:59.546005 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:59.547867 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.548360 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.548364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.548367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.548450 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.560176 (+ 11726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.560179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.560180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.560418 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:59.560529 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:59.561236 (+ 707us) server_negotiation.cc:299] Negotiation successful
0619 14:55:59.561348 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:55:59.564040 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:55:59.534123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55570 (local address 127.19.34.124:46409)
0619 14:55:59.534226 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.534229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.536820 (+ 2591us) server_negotiation.cc:407] Connection header received
0619 14:55:59.537592 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.537595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.537634 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.537679 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.540284 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.540962 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.544057 (+ 3095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:55:59.544402 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:55:59.544466 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:55:59.548587 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:55:59.548590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:55:59.548593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:55:59.548614 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:55:59.550332 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.560937 (+ 10605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.560940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.560942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.561016 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:55:59.563131 (+ 2115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:55:59.563134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:55:59.563136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:55:59.563350 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:55:59.563451 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:55:59.563805 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 14:55:59.563911 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:55:59.574453 9652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16239ee1b31c4b03a58943320c79032e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16239ee1b31c4b03a58943320c79032e" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33233 } } peers { permanent_uuid: "519442b09558472fa92037ef77a7b6f2" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } } peers { permanent_uuid: "08db7533e1114fc280efe6f3f3dd065d" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } } }
I20260619 14:55:59.574587 9652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:59.575651 9606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08db7533e1114fc280efe6f3f3dd065d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16239ee1b31c4b03a58943320c79032e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16239ee1b31c4b03a58943320c79032e" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33233 } } peers { permanent_uuid: "519442b09558472fa92037ef77a7b6f2" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } } peers { permanent_uuid: "08db7533e1114fc280efe6f3f3dd065d" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } } }
I20260619 14:55:59.575752 9606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08db7533e1114fc280efe6f3f3dd065d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:55:59.576851 9634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16239ee1b31c4b03a58943320c79032e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16239ee1b31c4b03a58943320c79032e" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33233 } } peers { permanent_uuid: "519442b09558472fa92037ef77a7b6f2" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } } peers { permanent_uuid: "08db7533e1114fc280efe6f3f3dd065d" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } } }
I20260619 14:55:59.576943 9634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e [sys.catalog]: This master's current role is: LEADER
I20260619 14:55:59.579698 9620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519442b09558472fa92037ef77a7b6f2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16239ee1b31c4b03a58943320c79032e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16239ee1b31c4b03a58943320c79032e" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 33233 } } peers { permanent_uuid: "519442b09558472fa92037ef77a7b6f2" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 34579 } } peers { permanent_uuid: "08db7533e1114fc280efe6f3f3dd065d" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 46409 } } }
I20260619 14:55:59.579793 9620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519442b09558472fa92037ef77a7b6f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:55:59.580461 9660 mvcc.cc:205] Tried to move back new op lower bound from 7298584410316787712 to 7298584410243506176. Current Snapshot: MvccSnapshot[applied={T|T < 7298584410316787712}]
I20260619 14:55:59.580641 9658 mvcc.cc:205] Tried to move back new op lower bound from 7298584410316787712 to 7298584410243506176. Current Snapshot: MvccSnapshot[applied={T|T < 7298584410316787712}]
I20260619 14:55:59.584636 9654 catalog_manager.cc:1347] Generated new cluster ID: 7ceb0dfa460f496495578a50b9e005b2
I20260619 14:55:59.584710 9654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:55:59.606137 9654 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:55:59.607259 9654 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:55:59.617756 9654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16239ee1b31c4b03a58943320c79032e: Generated new TSK 0
I20260619 14:55:59.618432 9654 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:55:59.649026 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:59.646060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37632 (local address 127.19.34.126:33233)
0619 14:55:59.646204 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.646208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.648019 (+ 1811us) server_negotiation.cc:407] Connection header received
0619 14:55:59.648212 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.648216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.648262 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.648335 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.648889 (+ 554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37632: BlockingRecv error: recv got EOF from 127.0.0.1:37632 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 14:55:59.650357 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:59.646821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52678 (local address 127.19.34.125:34579)
0619 14:55:59.646933 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.646937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.649581 (+ 2644us) server_negotiation.cc:407] Connection header received
0619 14:55:59.649725 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.649728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.649770 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.649813 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.650256 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52678: BlockingRecv error: recv got EOF from 127.0.0.1:52678 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260619 14:55:59.651597 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:59.647444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55576 (local address 127.19.34.124:46409)
0619 14:55:59.647562 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.647566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.650854 (+ 3288us) server_negotiation.cc:407] Connection header received
0619 14:55:59.650992 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.650995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.651035 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.651077 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.651497 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55576: BlockingRecv error: recv got EOF from 127.0.0.1:55576 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 14:55:59.655691 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:59.654466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52684 (local address 127.19.34.125:34579)
0619 14:55:59.654571 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.654577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.654798 (+ 221us) server_negotiation.cc:407] Connection header received
0619 14:55:59.655013 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.655016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.655086 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.655136 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.655601 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52684: BlockingRecv error: recv got EOF from 127.0.0.1:52684 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260619 14:55:59.660212 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:59.659001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52696 (local address 127.19.34.125:34579)
0619 14:55:59.659079 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.659082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.659510 (+ 428us) server_negotiation.cc:407] Connection header received
0619 14:55:59.659653 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.659655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.659697 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.659738 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.660127 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52696: BlockingRecv error: recv got EOF from 127.0.0.1:52696 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260619 14:55:59.669759 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:59.668622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52708 (local address 127.19.34.125:34579)
0619 14:55:59.668715 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.668719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.668941 (+ 222us) server_negotiation.cc:407] Connection header received
0619 14:55:59.669113 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.669116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.669173 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.669222 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.669672 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52708: BlockingRecv error: recv got EOF from 127.0.0.1:52708 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260619 14:55:59.677338 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:59.675730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52724 (local address 127.19.34.125:34579)
0619 14:55:59.675835 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.675839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.676588 (+ 749us) server_negotiation.cc:407] Connection header received
0619 14:55:59.676708 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.676710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.676757 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.676832 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.677236 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52724: BlockingRecv error: recv got EOF from 127.0.0.1:52724 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 14:55:59.688174 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:59.687171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52730 (local address 127.19.34.125:34579)
0619 14:55:59.687286 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.687292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.687504 (+ 212us) server_negotiation.cc:407] Connection header received
0619 14:55:59.687576 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.687578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.687626 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.687706 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.688077 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52730: BlockingRecv error: recv got EOF from 127.0.0.1:52730 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260619 14:55:59.706816 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:59.705764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52732 (local address 127.19.34.125:34579)
0619 14:55:59.705892 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.705899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.706059 (+ 160us) server_negotiation.cc:407] Connection header received
0619 14:55:59.706217 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.706219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.706258 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.706313 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.706721 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52732: BlockingRecv error: recv got EOF from 127.0.0.1:52732 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 14:55:59.741552 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:59.740590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.19.34.125:34579)
0619 14:55:59.740713 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.740718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.740847 (+ 129us) server_negotiation.cc:407] Connection header received
0619 14:55:59.740935 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.740938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.740977 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.741038 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.741439 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52748: BlockingRecv error: recv got EOF from 127.0.0.1:52748 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 14:55:59.808553 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:59.807443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.19.34.125:34579)
0619 14:55:59.807582 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.807599 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.807746 (+ 147us) server_negotiation.cc:407] Connection header received
0619 14:55:59.807887 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.807890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.807938 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.808000 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.808427 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52760: BlockingRecv error: recv got EOF from 127.0.0.1:52760 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 14:55:59.939479 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:55:59.938443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52772 (local address 127.19.34.125:34579)
0619 14:55:59.938582 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:55:59.938587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:55:59.938736 (+ 149us) server_negotiation.cc:407] Connection header received
0619 14:55:59.938840 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:55:59.938843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:55:59.938889 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:55:59.938957 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:55:59.939365 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52772: BlockingRecv error: recv got EOF from 127.0.0.1:52772 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 14:56:00.198589 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:56:00.197599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.19.34.125:34579)
0619 14:56:00.197733 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:00.197739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:00.197795 (+ 56us) server_negotiation.cc:407] Connection header received
0619 14:56:00.197936 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:00.197939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:00.198014 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:00.198096 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:00.198496 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52780: BlockingRecv error: recv got EOF from 127.0.0.1:52780 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:56:00.466641 9617 catalog_manager.cc:1259] Loaded cluster ID: 7ceb0dfa460f496495578a50b9e005b2
I20260619 14:56:00.466719 9617 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 08db7533e1114fc280efe6f3f3dd065d: loading cluster ID for follower catalog manager: success
I20260619 14:56:00.468125 9617 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 08db7533e1114fc280efe6f3f3dd065d: acquiring CA information for follower catalog manager: success
I20260619 14:56:00.469201 9617 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 08db7533e1114fc280efe6f3f3dd065d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:56:00.487607 9631 catalog_manager.cc:1259] Loaded cluster ID: 7ceb0dfa460f496495578a50b9e005b2
I20260619 14:56:00.487664 9631 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 519442b09558472fa92037ef77a7b6f2: loading cluster ID for follower catalog manager: success
I20260619 14:56:00.489153 9631 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 519442b09558472fa92037ef77a7b6f2: acquiring CA information for follower catalog manager: success
I20260619 14:56:00.490103 9631 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 519442b09558472fa92037ef77a7b6f2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:56:00.716063 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:00.712622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52794 (local address 127.19.34.125:34579)
0619 14:56:00.712831 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:00.712867 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:00.712971 (+ 104us) server_negotiation.cc:407] Connection header received
0619 14:56:00.713089 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:00.713092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:00.713230 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:00.713328 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:56:00.713531 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:00.714007 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:00.714719 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:00.714903 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:00.714966 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:00.715216 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:56:00.715622 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:56:00.715736 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:00.715825 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 14:56:00.715870 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260619 14:56:00.721922 9736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:00.718771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55588 (local address 127.19.34.124:46409)
0619 14:56:00.719003 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:00.719006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:00.719018 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:56:00.719132 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:00.719134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:00.719232 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:00.719332 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:56:00.719508 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:00.720021 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:00.720818 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:00.721008 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:00.721075 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:00.721323 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:56:00.721665 (+ 342us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:56:00.721742 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:00.721788 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 14:56:00.721828 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":80,"threads_started":1}
I20260619 14:56:00.727877 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:00.724356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52796 (local address 127.19.34.125:34579)
0619 14:56:00.724450 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:00.724454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:00.724631 (+ 177us) server_negotiation.cc:407] Connection header received
0619 14:56:00.724814 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:00.724817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:00.724880 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:00.724927 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:56:00.725310 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:00.726058 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:00.726979 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:00.727165 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:00.727259 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:00.727474 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:56:00.727527 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:56:00.727576 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:00.727700 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 14:56:00.727742 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:56:00.728379 9736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:00.724684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55604 (local address 127.19.34.124:46409)
0619 14:56:00.724806 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:00.724809 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:00.725155 (+ 346us) server_negotiation.cc:407] Connection header received
0619 14:56:00.725255 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:00.725259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:00.725314 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:00.725399 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:56:00.725622 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:00.726087 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:00.727331 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:00.727579 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:00.727678 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:00.727966 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:56:00.728030 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:56:00.728107 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:00.728221 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 14:56:00.728265 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:56:00.729532 9743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:00.724328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37634 (local address 127.19.34.126:33233)
0619 14:56:00.724577 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:00.724581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:00.724614 (+ 33us) server_negotiation.cc:407] Connection header received
0619 14:56:00.724691 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:00.724693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:00.724869 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:00.725000 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:56:00.725380 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:00.726161 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:00.727734 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:00.728415 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:00.728668 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:00.728940 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:56:00.729195 (+ 255us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:56:00.729285 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:00.729353 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 14:56:00.729399 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":67,"threads_started":1}
I20260619 14:56:00.731912 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 9381
I20260619 14:56:00.737709 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 9456
I20260619 14:56:00.742630 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 9531
2026-06-19T14:56:00Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2920 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:56:00 dist-test-slave-3qdh krb5kdc[9747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:56:00 dist-test-slave-3qdh krb5kdc[9747](info): set up 2 sockets
Jun 19 14:56:00 dist-test-slave-3qdh krb5kdc[9747](info): commencing operation
krb5kdc: starting...
W20260619 14:56:01.704159 19593 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/test-user.keytab.
Jun 19 14:56:01 dist-test-slave-3qdh krb5kdc[9747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:56:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:56:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.34.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
Entry for principal kudu/127.19.34.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab.
I20260619 14:56:01.813076 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:43065
--webserver_interface=127.19.34.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:41925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:43065,127.19.34.125:43051,127.19.34.124:43245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:56:01.910712 9760 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:56:01.910921 9760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:56:01.910944 9760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:56:01.910977 9760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:56:01.914359 9760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:56:01.914403 9760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:56:01.914423 9760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:56:01.914441 9760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:56:01.914458 9760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:56:01.920073 9760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:41925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:43065,127.19.34.125:43051,127.19.34.124:43245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.126.keytab
--principal=kudu/127.19.34.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.126:43065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.34.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:56:01.920926 9760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:56:01.921729 9760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:56:01.929059 9768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:56:01.929114 9766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:56:01.929078 9765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:56:01.929855 9760 server_base.cc:1034] running on GCE node
I20260619 14:56:01.930436 9760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:56:01.931391 9760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:56:01.932565 9760 hybrid_clock.cc:648] HybridClock initialized: now 1781880961932556 us; error 43 us; skew 500 ppm
Jun 19 14:56:01 dist-test-slave-3qdh krb5kdc[9747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880961, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:56:01.934935 9760 init.cc:377] Logged in from keytab as kudu/127.19.34.126@KRBTEST.COM (short username kudu)
I20260619 14:56:01.935806 9760 webserver.cc:460] Webserver started at http://127.19.34.126:34883/ using document root <none> and password file <none>
I20260619 14:56:01.936261 9760 fs_manager.cc:359] Metadata directory not provided
I20260619 14:56:01.936393 9760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:56:01.936591 9760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:56:01.938233 9760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-0/data/instance:
uuid: "bc103509eebc4deea2e6b182a14a8196"
format_stamp: "Formatted at 2026-06-19 14:56:01 on dist-test-slave-3qdh"
server_key: "055c1b09e8f2bb76ec47218b8cbd3ee1"
server_key_iv: "b5b2c817857c6ec8aa9a50fd29a0a887"
server_key_version: "encryptionkey@0"
I20260619 14:56:01.938696 9760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-0/wal/instance:
uuid: "bc103509eebc4deea2e6b182a14a8196"
format_stamp: "Formatted at 2026-06-19 14:56:01 on dist-test-slave-3qdh"
server_key: "055c1b09e8f2bb76ec47218b8cbd3ee1"
server_key_iv: "b5b2c817857c6ec8aa9a50fd29a0a887"
server_key_version: "encryptionkey@0"
I20260619 14:56:01.941972 9760 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260619 14:56:01.944114 9775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:56:01.945290 9760 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:56:01.945384 9760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-0/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-0/wal
uuid: "bc103509eebc4deea2e6b182a14a8196"
format_stamp: "Formatted at 2026-06-19 14:56:01 on dist-test-slave-3qdh"
server_key: "055c1b09e8f2bb76ec47218b8cbd3ee1"
server_key_iv: "b5b2c817857c6ec8aa9a50fd29a0a887"
server_key_version: "encryptionkey@0"
I20260619 14:56:01.945461 9760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:56:01.964707 9760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:56:01.967212 9760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:56:01.967335 9760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:56:01.973853 9760 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.126:43065
I20260619 14:56:01.974174 9827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.126:43065 every 8 connection(s)
I20260619 14:56:01.974691 9760 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-0/data/info.pb
I20260619 14:56:01.977337 9828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:56:01.977806 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 9760
I20260619 14:56:01.977926 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-0/wal/instance
I20260619 14:56:01.978240 19593 external_mini_cluster.cc:1428] Setting key 2f763123c2d8915cc66d0ba1a69714cb
I20260619 14:56:01.981952 9828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43065 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.19.34.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.125@KRBTEST.COM" created.
Jun 19 14:56:01 dist-test-slave-3qdh krb5kdc[9747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880961, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:56:01.996670 9832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:01.984090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43065 (local address 127.0.0.1:53714)
0619 14:56:01.984636 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:01.984668 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 14:56:01.985462 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:56:01.986152 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:56:01.986157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:56:01.986610 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:56:01.986864 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:56:01.986875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:56:01.988058 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:56:01.988061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:56:01.989213 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:56:01.989226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:56:01.989806 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:56:01.989809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:56:01.989895 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:01.990417 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:56:01.990458 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:56:01.991966 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:56:01.994073 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:56:01.994080 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:56:01.994086 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:56:01.994289 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:56:01.994634 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:56:01.994636 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:56:01.994637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:56:01.994726 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:56:01.995441 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:56:01.995445 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:56:01.995551 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 14:56:01.995691 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 14:56:01.995895 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"spinlock_wait_cycles":896,"thread_start_us":163,"threads_started":1}
I20260619 14:56:01.996668 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:01.984105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.19.34.126:43065)
0619 14:56:01.984485 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:01.984502 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:01.984765 (+ 263us) server_negotiation.cc:407] Connection header received
0619 14:56:01.985755 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:01.985759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:01.985954 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:01.986167 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:01.987002 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:01.987923 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:01.989333 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:01.989664 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:01.989823 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:01.992106 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:56:01.992115 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:56:01.992129 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:56:01.992158 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:56:01.993972 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:01.994410 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:01.994418 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:01.994421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:01.994542 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:01.994813 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:01.994816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:01.994818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:01.995316 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:56:01.995465 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:01.995802 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 14:56:01.995952 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":123,"threads_started":1}
I20260619 14:56:01.998543 9828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:56:01.999143 9832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:01.998960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:43051 (local address 127.0.0.1:59334)
0619 14:56:01.999050 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:01.999102 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.125:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 14:56:01.999300 9779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.34.125:43051: connect: Connection refused (error 111)
I20260619 14:56:01.999951 9832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:01.999795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:43051 (local address 127.0.0.1:59340)
0619 14:56:01.999878 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:01.999913 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.125:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 14:56:02.000067 9828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.125:43051: Network error: Client connection negotiation failed: client connection to 127.19.34.125:43051: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.125.keytab.
Entry for principal kudu/127.19.34.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.125.keytab.
I20260619 14:56:02.002875 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.125:43051
--webserver_interface=127.19.34.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.125.keytab
--principal=kudu/127.19.34.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:41925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:43065,127.19.34.125:43051,127.19.34.124:43245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:56:02.049273 9828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } attempt: 1
I20260619 14:56:02.050058 9832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.049753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:43051 (local address 127.0.0.1:59346)
0619 14:56:02.049915 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.049978 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.125:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 14:56:02.050769 9832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.050548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:43051 (local address 127.0.0.1:59360)
0619 14:56:02.050641 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.050711 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.125:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 14:56:02.050905 9828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.125:43051: Network error: Client connection negotiation failed: client connection to 127.19.34.125:43051: connect: Connection refused (error 111)
W20260619 14:56:02.099740 9835 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:56:02.099937 9835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:56:02.099957 9835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:56:02.099987 9835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:56:02.103338 9835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:56:02.103379 9835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:56:02.103415 9835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:56:02.103430 9835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:56:02.103442 9835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:56:02.107923 9835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:41925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:43065,127.19.34.125:43051,127.19.34.124:43245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.125.keytab
--principal=kudu/127.19.34.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.125:43051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.34.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:56:02.108739 9835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:56:02.109498 9835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:56:02.116532 9840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:56:02.116544 9841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:56:02.116547 9843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:56:02.116811 9835 server_base.cc:1034] running on GCE node
I20260619 14:56:02.117291 9835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:56:02.118235 9835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:56:02.119045 9828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } attempt: 2
I20260619 14:56:02.119402 9835 hybrid_clock.cc:648] HybridClock initialized: now 1781880962119389 us; error 26 us; skew 500 ppm
I20260619 14:56:02.119664 9832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.119425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:43051 (local address 127.0.0.1:59362)
0619 14:56:02.119541 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.119602 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.125:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 14:56:02.120419 9832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.120124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:43051 (local address 127.0.0.1:59368)
0619 14:56:02.120265 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.120341 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.125:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 14:56:02.120566 9828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.125:43051: Network error: Client connection negotiation failed: client connection to 127.19.34.125:43051: connect: Connection refused (error 111)
Jun 19 14:56:02 dist-test-slave-3qdh krb5kdc[9747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880962, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:56:02.122002 9835 init.cc:377] Logged in from keytab as kudu/127.19.34.125@KRBTEST.COM (short username kudu)
I20260619 14:56:02.122779 9835 webserver.cc:460] Webserver started at http://127.19.34.125:40001/ using document root <none> and password file <none>
I20260619 14:56:02.123211 9835 fs_manager.cc:359] Metadata directory not provided
I20260619 14:56:02.123262 9835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:56:02.123451 9835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:56:02.125181 9835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-1/data/instance:
uuid: "03aac2d3dc9a4b478f0cf7459f7e63db"
format_stamp: "Formatted at 2026-06-19 14:56:02 on dist-test-slave-3qdh"
server_key: "2673547ea8579e0db35eb05db84a3114"
server_key_iv: "983e5de6d2923de959fcf57b1bbffc26"
server_key_version: "encryptionkey@0"
I20260619 14:56:02.125715 9835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-1/wal/instance:
uuid: "03aac2d3dc9a4b478f0cf7459f7e63db"
format_stamp: "Formatted at 2026-06-19 14:56:02 on dist-test-slave-3qdh"
server_key: "2673547ea8579e0db35eb05db84a3114"
server_key_iv: "983e5de6d2923de959fcf57b1bbffc26"
server_key_version: "encryptionkey@0"
I20260619 14:56:02.128980 9835 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:56:02.130947 9850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:56:02.131980 9835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 14:56:02.132100 9835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-1/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-1/wal
uuid: "03aac2d3dc9a4b478f0cf7459f7e63db"
format_stamp: "Formatted at 2026-06-19 14:56:02 on dist-test-slave-3qdh"
server_key: "2673547ea8579e0db35eb05db84a3114"
server_key_iv: "983e5de6d2923de959fcf57b1bbffc26"
server_key_version: "encryptionkey@0"
I20260619 14:56:02.132194 9835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:56:02.141566 9835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:56:02.143702 9835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:56:02.143857 9835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:56:02.149781 9835 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.125:43051
I20260619 14:56:02.150007 9902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.125:43051 every 8 connection(s)
I20260619 14:56:02.150522 9835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-1/data/info.pb
I20260619 14:56:02.152808 9903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:56:02.155956 9903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43065 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:56:02.157404 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 9835
I20260619 14:56:02.157512 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-1/wal/instance
I20260619 14:56:02.157778 19593 external_mini_cluster.cc:1428] Setting key 0c597e54827db42799749a7792601b3e
Jun 19 14:56:02 dist-test-slave-3qdh krb5kdc[9747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880962, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.125@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.34.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.34.124@KRBTEST.COM" created.
I20260619 14:56:02.169524 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.157495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53730 (local address 127.19.34.126:43065)
0619 14:56:02.157627 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.157631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.158496 (+ 865us) server_negotiation.cc:407] Connection header received
0619 14:56:02.159668 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.159674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.159731 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.159820 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.160998 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.161447 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.163041 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.163238 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.163307 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.165612 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:56:02.165616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:56:02.165618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:56:02.165639 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:56:02.167467 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.168054 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.168060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.168063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.168147 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.168509 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.168514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.168516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.168687 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:56:02.168792 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:02.169253 (+ 461us) server_negotiation.cc:299] Negotiation successful
0619 14:56:02.169367 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:56:02.170115 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.157884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43065 (local address 127.0.0.1:53730)
0619 14:56:02.158343 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.158398 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:56:02.159488 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:56:02.159940 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:56:02.159946 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:56:02.160250 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:56:02.160865 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:56:02.160883 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.161585 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.161589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:56:02.162934 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:56:02.162946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.163369 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.163373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:56:02.163559 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.164031 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:56:02.164052 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:56:02.165440 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:56:02.167579 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:56:02.167585 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:56:02.167597 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:56:02.167956 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:56:02.168243 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:56:02.168248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:56:02.168250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:56:02.168415 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:56:02.168791 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:56:02.168797 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:56:02.169015 (+ 218us) client_negotiation.cc:769] Sending connection context
0619 14:56:02.169187 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 14:56:02.169470 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":131,"threads_started":1}
I20260619 14:56:02.171182 9903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:56:02 dist-test-slave-3qdh krb5kdc[9747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880962, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.125@KRBTEST.COM for kudu/127.19.34.125@KRBTEST.COM
I20260619 14:56:02.180436 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.171559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:43051 (local address 127.0.0.1:59376)
0619 14:56:02.171647 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.171660 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:56:02.171720 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:56:02.172206 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:56:02.172208 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:56:02.172368 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:56:02.172446 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:56:02.172450 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.173397 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.173403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:56:02.173933 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:56:02.173941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.174519 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.174523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:56:02.174616 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.175037 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:56:02.175052 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:56:02.176576 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:56:02.178687 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:56:02.178692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:56:02.178694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:56:02.178964 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:56:02.179273 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:56:02.179279 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:56:02.179282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:56:02.179378 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:56:02.179930 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:56:02.179933 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:56:02.180019 (+ 86us) client_negotiation.cc:769] Sending connection context
0619 14:56:02.180142 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 14:56:02.180260 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260619 14:56:02.180600 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.171707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59376 (local address 127.19.34.125:43051)
0619 14:56:02.171945 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.171952 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.171966 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:56:02.171991 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.171994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.172108 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.172224 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.172536 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.173169 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.174056 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.174402 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.174513 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.176710 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:56:02.176727 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:56:02.176736 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:56:02.176756 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:56:02.178545 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.179084 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.179090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.179092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.179177 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.179474 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.179480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.179483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.179719 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:56:02.179959 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:02.180272 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 14:56:02.180407 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.34.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.124.keytab.
Entry for principal kudu/127.19.34.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.124.keytab.
I20260619 14:56:02.182019 9903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:56:02.182554 19593 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0Qcekt/build/debug/bin/kudu
/tmp/dist-test-task0Qcekt/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.124:43245
--webserver_interface=127.19.34.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.124.keytab
--principal=kudu/127.19.34.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.34.84:41925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.34.126:43065,127.19.34.125:43051,127.19.34.124:43245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:56:02.182618 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.182361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:43245 (local address 127.0.0.1:34894)
0619 14:56:02.182473 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.182549 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:43245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 14:56:02.182842 9852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.34.124:43245: connect: Connection refused (error 111)
I20260619 14:56:02.183547 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.183369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:43245 (local address 127.0.0.1:34900)
0619 14:56:02.183460 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.183505 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:43245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 14:56:02.183701 9903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.124:43245: Network error: Client connection negotiation failed: client connection to 127.19.34.124:43245: connect: Connection refused (error 111)
I20260619 14:56:02.211727 9828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } attempt: 3
Jun 19 14:56:02 dist-test-slave-3qdh krb5kdc[9747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880961, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for kudu/127.19.34.125@KRBTEST.COM
I20260619 14:56:02.220062 9832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.212202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:43051 (local address 127.0.0.1:59388)
0619 14:56:02.212336 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.212351 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:56:02.212428 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:56:02.212678 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:56:02.212680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:56:02.212809 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:56:02.212897 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:56:02.212901 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.213861 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.213863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:56:02.214365 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:56:02.214371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.214978 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.214980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:56:02.215058 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.215533 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:56:02.215544 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:56:02.216928 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:56:02.218842 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:56:02.218845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:56:02.218846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:56:02.219010 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:56:02.219283 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:56:02.219285 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:56:02.219286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:56:02.219348 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:56:02.219728 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:56:02.219730 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:56:02.219795 (+ 65us) client_negotiation.cc:769] Sending connection context
0619 14:56:02.219864 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 14:56:02.219961 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 14:56:02.220191 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.212273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59388 (local address 127.19.34.125:43051)
0619 14:56:02.212407 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.212411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.212429 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:56:02.212494 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.212497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.212539 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.212652 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.213010 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.213759 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.214479 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.214868 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.214987 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.217114 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:56:02.217118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:56:02.217121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:56:02.217153 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:56:02.218739 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.219101 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.219105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.219107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.219194 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.219428 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.219431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.219433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.219642 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:56:02.219742 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:02.219949 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 14:56:02.220052 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:56:02.221180 9828 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:56:02.221843 9832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.221620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:43245 (local address 127.0.0.1:34906)
0619 14:56:02.221740 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.221796 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:43245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 14:56:02.222512 9832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.222319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:43245 (local address 127.0.0.1:34910)
0619 14:56:02.222438 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.222473 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:43245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260619 14:56:02.222703 9828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.124:43245: Network error: Client connection negotiation failed: client connection to 127.19.34.124:43245: connect: Connection refused (error 111)
I20260619 14:56:02.232946 9903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } attempt: 1
I20260619 14:56:02.233448 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.233256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:43245 (local address 127.0.0.1:34912)
0619 14:56:02.233352 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.233400 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:43245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 14:56:02.234091 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.233925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:43245 (local address 127.0.0.1:34922)
0619 14:56:02.234010 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.234048 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:43245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260619 14:56:02.234275 9903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.124:43245: Network error: Client connection negotiation failed: client connection to 127.19.34.124:43245: connect: Connection refused (error 111)
I20260619 14:56:02.253865 9828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } attempt: 1
I20260619 14:56:02.254426 9832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.254215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:43245 (local address 127.0.0.1:34930)
0619 14:56:02.254321 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.254374 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:43245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 14:56:02.255074 9832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.254886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:43245 (local address 127.0.0.1:34938)
0619 14:56:02.254981 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.255023 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:43245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 14:56:02.255270 9828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.124:43245: Network error: Client connection negotiation failed: client connection to 127.19.34.124:43245: connect: Connection refused (error 111)
W20260619 14:56:02.286213 9910 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:56:02.286432 9910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:56:02.286454 9910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:56:02.286487 9910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:56:02.290235 9910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:56:02.290277 9910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:56:02.290293 9910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:56:02.290306 9910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:56:02.290318 9910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:56:02.294828 9910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.34.84:41925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.34.126:43065,127.19.34.125:43051,127.19.34.124:43245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/krb5kdc/kudu_127.19.34.124.keytab
--principal=kudu/127.19.34.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.34.124:43245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.34.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:56:02.295603 9910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:56:02.296427 9910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:56:02.302450 9903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } attempt: 2
I20260619 14:56:02.303236 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.302942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:43245 (local address 127.0.0.1:34950)
0619 14:56:02.303077 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.303145 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:43245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 14:56:02.303356 9915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:56:02.303416 9916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:56:02.303436 9918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:56:02.303716 9910 server_base.cc:1034] running on GCE node
I20260619 14:56:02.303993 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.303795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:43245 (local address 127.0.0.1:34954)
0619 14:56:02.303913 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.303945 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:43245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260619 14:56:02.304189 9903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.124:43245: Network error: Client connection negotiation failed: client connection to 127.19.34.124:43245: connect: Connection refused (error 111)
I20260619 14:56:02.304214 9910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:56:02.305094 9910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:56:02.306241 9910 hybrid_clock.cc:648] HybridClock initialized: now 1781880962306229 us; error 21 us; skew 500 ppm
Jun 19 14:56:02 dist-test-slave-3qdh krb5kdc[9747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880962, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:56:02.308552 9910 init.cc:377] Logged in from keytab as kudu/127.19.34.124@KRBTEST.COM (short username kudu)
I20260619 14:56:02.309324 9910 webserver.cc:460] Webserver started at http://127.19.34.124:38403/ using document root <none> and password file <none>
I20260619 14:56:02.309777 9910 fs_manager.cc:359] Metadata directory not provided
I20260619 14:56:02.309829 9910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:56:02.310020 9910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:56:02.311640 9910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-2/data/instance:
uuid: "6e22926006444dd7ba9e5cac38cc1739"
format_stamp: "Formatted at 2026-06-19 14:56:02 on dist-test-slave-3qdh"
server_key: "ffcef34c3c5802aa1848b134817427ec"
server_key_iv: "3aff6172c020ee3420564bdb4091afe8"
server_key_version: "encryptionkey@0"
I20260619 14:56:02.312089 9910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-2/wal/instance:
uuid: "6e22926006444dd7ba9e5cac38cc1739"
format_stamp: "Formatted at 2026-06-19 14:56:02 on dist-test-slave-3qdh"
server_key: "ffcef34c3c5802aa1848b134817427ec"
server_key_iv: "3aff6172c020ee3420564bdb4091afe8"
server_key_version: "encryptionkey@0"
I20260619 14:56:02.315505 9910 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:56:02.317732 9925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:56:02.318864 9910 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 14:56:02.318990 9910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-2/data,/tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-2/wal
uuid: "6e22926006444dd7ba9e5cac38cc1739"
format_stamp: "Formatted at 2026-06-19 14:56:02 on dist-test-slave-3qdh"
server_key: "ffcef34c3c5802aa1848b134817427ec"
server_key_iv: "3aff6172c020ee3420564bdb4091afe8"
server_key_version: "encryptionkey@0"
I20260619 14:56:02.319079 9910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:56:02.330427 9828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } attempt: 2
I20260619 14:56:02.331041 9832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.330793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:43245 (local address 127.0.0.1:34970)
0619 14:56:02.330905 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.330965 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:43245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 14:56:02.331701 9832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.331508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:43245 (local address 127.0.0.1:34982)
0619 14:56:02.331605 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.331648 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.34.124:43245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 14:56:02.331843 9828 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.34.124:43245: Network error: Client connection negotiation failed: client connection to 127.19.34.124:43245: connect: Connection refused (error 111)
I20260619 14:56:02.333455 9910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:56:02.336097 9910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:56:02.336283 9910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:56:02.342118 9910 rpc_server.cc:307] RPC server started. Bound to: 127.19.34.124:43245
I20260619 14:56:02.342351 9977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.34.124:43245 every 8 connection(s)
I20260619 14:56:02.342922 9910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-2/data/info.pb
I20260619 14:56:02.345340 9978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:56:02.347486 19593 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0Qcekt/build/debug/bin/kudu as pid 9910
I20260619 14:56:02.347596 19593 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0Qcekt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880697016847-19593-0/minicluster-data/master-2/wal/instance
I20260619 14:56:02.347923 19593 external_mini_cluster.cc:1428] Setting key d5e4d9661672288032629b1eab5e0dc6
I20260619 14:56:02.349066 9978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43065 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:56:02 dist-test-slave-3qdh krb5kdc[9747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880962, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.124@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
Jun 19 14:56:02 dist-test-slave-3qdh krb5kdc[9747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.124@KRBTEST.COM
I20260619 14:56:02.359180 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.350568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53736 (local address 127.19.34.126:43065)
0619 14:56:02.350715 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.350720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.351339 (+ 619us) server_negotiation.cc:407] Connection header received
0619 14:56:02.351497 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.351500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.351535 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.351619 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.352209 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.352749 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.353653 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.353852 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.353948 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.356032 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:56:02.356039 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:56:02.356041 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:56:02.356071 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:56:02.357338 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.357803 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.357806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.357808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.357883 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.358146 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.358151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.358154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.358356 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:56:02.358433 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:02.358989 (+ 556us) server_negotiation.cc:299] Negotiation successful
0619 14:56:02.359093 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:56:02.359939 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.350511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.126:43065 (local address 127.0.0.1:53736)
0619 14:56:02.351237 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.351266 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 14:56:02.351386 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:56:02.351660 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:56:02.351665 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:56:02.351955 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:56:02.352081 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:56:02.352089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.352872 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.352875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:56:02.353540 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:56:02.353548 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.353976 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.353980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:56:02.354094 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.354509 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:56:02.354533 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:56:02.355887 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:56:02.357428 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:56:02.357437 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:56:02.357446 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:56:02.357719 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:56:02.357964 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:56:02.357966 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:56:02.357967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:56:02.358066 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:56:02.358494 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:56:02.358499 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:56:02.358739 (+ 240us) client_negotiation.cc:769] Sending connection context
0619 14:56:02.358917 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 14:56:02.359150 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":21,"thread_start_us":156,"threads_started":1}
I20260619 14:56:02.359969 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.349142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34992 (local address 127.19.34.124:43245)
0619 14:56:02.349547 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.349561 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.349590 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:56:02.350150 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.350175 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.350495 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.350845 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.351577 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.352473 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.353274 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.353674 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.353817 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.356630 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:56:02.356634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:56:02.356653 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:56:02.356669 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:56:02.358229 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.358671 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.358676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.358677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.358755 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.359023 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.359025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.359026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.359242 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:56:02.359392 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:02.359689 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 14:56:02.359780 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":275,"spinlock_wait_cycles":768,"thread_start_us":104,"threads_started":1}
I20260619 14:56:02.360953 9978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:56:02 dist-test-slave-3qdh krb5kdc[9747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880962, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.124@KRBTEST.COM for kudu/127.19.34.125@KRBTEST.COM
Jun 19 14:56:02 dist-test-slave-3qdh krb5kdc[9747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.126@KRBTEST.COM
I20260619 14:56:02.368175 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.361351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.125:43051 (local address 127.0.0.1:59404)
0619 14:56:02.361470 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.361480 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:56:02.361538 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:56:02.361759 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:56:02.361762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:56:02.361894 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:56:02.362006 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:56:02.362014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.362763 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.362768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:56:02.363253 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:56:02.363261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.363619 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.363622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:56:02.363709 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.364086 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:56:02.364095 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:56:02.365108 (+ 1013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:56:02.366920 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:56:02.366924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:56:02.366926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:56:02.367096 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:56:02.367349 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:56:02.367351 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:56:02.367353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:56:02.367425 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:56:02.367749 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:56:02.367752 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:56:02.367845 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 14:56:02.367954 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 14:56:02.368051 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 14:56:02.368214 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.361450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59404 (local address 127.19.34.125:43051)
0619 14:56:02.361563 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.361569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.361583 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:56:02.361624 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.361626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.361667 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.361737 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.362155 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.362667 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.363355 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.363530 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.363593 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.365225 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:56:02.365228 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:56:02.365230 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:56:02.365276 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:56:02.366833 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.367184 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.367188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.367189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.367262 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.367505 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.367510 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.367511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.367664 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:56:02.367757 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:02.367998 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 14:56:02.368085 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:56:02.369027 9978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:56:02.369984 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.362031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53750 (local address 127.19.34.126:43065)
0619 14:56:02.362132 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.362135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.362146 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:56:02.362239 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.362241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.362273 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.362343 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.362788 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.363192 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.363991 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.364234 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.364352 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.366796 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:56:02.366798 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:56:02.366800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:56:02.366819 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:56:02.368043 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.368567 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.368571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.368573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.368665 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.368975 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.368978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.368980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.369183 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:56:02.369753 (+ 570us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:02.369819 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 14:56:02.369901 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Jun 19 14:56:02 dist-test-slave-3qdh krb5kdc[9747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880962, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.124@KRBTEST.COM for kudu/127.19.34.124@KRBTEST.COM
Jun 19 14:56:02 dist-test-slave-3qdh krb5kdc[9747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.34.125@KRBTEST.COM
I20260619 14:56:02.376478 9982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.369411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:43245 (local address 127.0.0.1:35000)
0619 14:56:02.369507 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.369517 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:56:02.369603 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:56:02.369804 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:56:02.369806 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:56:02.369939 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:56:02.369986 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:56:02.369990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.370669 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.370672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:56:02.371262 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:56:02.371270 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.371740 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.371743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:56:02.371802 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.372101 (+ 299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:56:02.372112 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:56:02.373353 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:56:02.375006 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:56:02.375009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:56:02.375012 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:56:02.375266 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:56:02.375519 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:56:02.375524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:56:02.375526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:56:02.375616 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:56:02.375984 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:56:02.375987 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:56:02.376067 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 14:56:02.376177 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 14:56:02.376311 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 14:56:02.376693 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.369527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35000 (local address 127.19.34.124:43245)
0619 14:56:02.369641 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.369646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.369665 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:56:02.369689 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.369691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.369719 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.369795 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.370111 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.370571 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.371355 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.371561 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.371664 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.373463 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:56:02.373465 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:56:02.373467 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:56:02.373484 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:56:02.374895 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.375367 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.375369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.375370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.375433 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.375726 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.375732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.375734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.375886 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:56:02.375984 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:02.376218 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 14:56:02.376414 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":55}
I20260619 14:56:02.379006 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.370942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59406 (local address 127.19.34.125:43051)
0619 14:56:02.371034 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.371037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.371049 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:56:02.371082 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.371084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.371115 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.371197 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.371644 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.372241 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.373240 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.373535 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.373601 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.375985 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:56:02.375989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:56:02.375992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:56:02.376014 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:56:02.377546 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.377872 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.377875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.377876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.377947 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.378177 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.378181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.378183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.378364 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:56:02.378609 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:02.378680 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 14:56:02.378877 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:56:02.379838 9978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e22926006444dd7ba9e5cac38cc1739: Bootstrap starting.
I20260619 14:56:02.381851 19593 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:56:02.382162 9978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e22926006444dd7ba9e5cac38cc1739: Neither blocks nor log segments found. Creating new log.
I20260619 14:56:02.382753 9978 log.cc:824] T 00000000000000000000000000000000 P 6e22926006444dd7ba9e5cac38cc1739: Log is configured to *not* fsync() on all Append() calls
I20260619 14:56:02.382822 9802 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bc103509eebc4deea2e6b182a14a8196"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53750
I20260619 14:56:02.384207 9978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e22926006444dd7ba9e5cac38cc1739: No bootstrap required, opened a new log
I20260619 14:56:02.386730 9978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e22926006444dd7ba9e5cac38cc1739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc103509eebc4deea2e6b182a14a8196" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43065 } } peers { permanent_uuid: "03aac2d3dc9a4b478f0cf7459f7e63db" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } } peers { permanent_uuid: "6e22926006444dd7ba9e5cac38cc1739" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } }
I20260619 14:56:02.386880 9978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e22926006444dd7ba9e5cac38cc1739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e22926006444dd7ba9e5cac38cc1739, State: Initialized, Role: FOLLOWER
I20260619 14:56:02.387154 9978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e22926006444dd7ba9e5cac38cc1739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc103509eebc4deea2e6b182a14a8196" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43065 } } peers { permanent_uuid: "03aac2d3dc9a4b478f0cf7459f7e63db" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } } peers { permanent_uuid: "6e22926006444dd7ba9e5cac38cc1739" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } }
I20260619 14:56:02.387598 9984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e22926006444dd7ba9e5cac38cc1739 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc103509eebc4deea2e6b182a14a8196" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43065 } } peers { permanent_uuid: "03aac2d3dc9a4b478f0cf7459f7e63db" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } } peers { permanent_uuid: "6e22926006444dd7ba9e5cac38cc1739" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } } }
I20260619 14:56:02.387718 9984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e22926006444dd7ba9e5cac38cc1739 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:56:02.388126 9978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e22926006444dd7ba9e5cac38cc1739 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:56:02.392333 9995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6e22926006444dd7ba9e5cac38cc1739: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:56:02.392418 9995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:56:02.391248 9910 master_runner.cc:186] Error getting master registration for 127.19.34.126:43065: OK, instance_id { permanent_uuid: "bc103509eebc4deea2e6b182a14a8196" instance_seqno: 1781880961972015 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:56:02.394192 9910 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:56:02.395361 9903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } attempt: 3
Jun 19 14:56:02 dist-test-slave-3qdh krb5kdc[9747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880962, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.125@KRBTEST.COM for kudu/127.19.34.124@KRBTEST.COM
I20260619 14:56:02.403589 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.395713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:43245 (local address 127.0.0.1:35012)
0619 14:56:02.395867 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.395881 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:56:02.395964 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:56:02.396216 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:56:02.396219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:56:02.396435 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:56:02.396529 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:56:02.396536 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.397411 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.397414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:56:02.398150 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:56:02.398171 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.398653 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.398657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:56:02.398733 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.399128 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:56:02.399141 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:56:02.400533 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:56:02.402189 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:56:02.402193 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:56:02.402195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:56:02.402400 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:56:02.402682 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:56:02.402686 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:56:02.402688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:56:02.402768 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:56:02.403123 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:56:02.403126 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:56:02.403215 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 14:56:02.403315 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 14:56:02.403435 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260619 14:56:02.403589 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.395782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35012 (local address 127.19.34.124:43245)
0619 14:56:02.395887 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.395891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.395924 (+ 33us) server_negotiation.cc:407] Connection header received
0619 14:56:02.396046 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.396051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.396087 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.396171 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.396641 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.397307 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.398261 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.398556 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.398632 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.400676 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:56:02.400679 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:56:02.400681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:56:02.400699 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:56:02.402070 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.402515 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.402519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.402521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.402591 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.402852 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.402857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.402859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.403023 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:56:02.403125 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:02.403362 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 14:56:02.403456 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:56:02.406380 9903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03aac2d3dc9a4b478f0cf7459f7e63db: Bootstrap starting.
I20260619 14:56:02.408722 9903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03aac2d3dc9a4b478f0cf7459f7e63db: Neither blocks nor log segments found. Creating new log.
I20260619 14:56:02.409489 9903 log.cc:824] T 00000000000000000000000000000000 P 03aac2d3dc9a4b478f0cf7459f7e63db: Log is configured to *not* fsync() on all Append() calls
I20260619 14:56:02.411355 9903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03aac2d3dc9a4b478f0cf7459f7e63db: No bootstrap required, opened a new log
I20260619 14:56:02.414822 9903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03aac2d3dc9a4b478f0cf7459f7e63db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc103509eebc4deea2e6b182a14a8196" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43065 } } peers { permanent_uuid: "03aac2d3dc9a4b478f0cf7459f7e63db" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } } peers { permanent_uuid: "6e22926006444dd7ba9e5cac38cc1739" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } }
I20260619 14:56:02.415015 9903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03aac2d3dc9a4b478f0cf7459f7e63db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03aac2d3dc9a4b478f0cf7459f7e63db, State: Initialized, Role: FOLLOWER
I20260619 14:56:02.415342 9903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03aac2d3dc9a4b478f0cf7459f7e63db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc103509eebc4deea2e6b182a14a8196" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43065 } } peers { permanent_uuid: "03aac2d3dc9a4b478f0cf7459f7e63db" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } } peers { permanent_uuid: "6e22926006444dd7ba9e5cac38cc1739" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } }
I20260619 14:56:02.415762 9998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03aac2d3dc9a4b478f0cf7459f7e63db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc103509eebc4deea2e6b182a14a8196" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43065 } } peers { permanent_uuid: "03aac2d3dc9a4b478f0cf7459f7e63db" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } } peers { permanent_uuid: "6e22926006444dd7ba9e5cac38cc1739" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } } }
I20260619 14:56:02.415902 9998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03aac2d3dc9a4b478f0cf7459f7e63db [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:56:02.416554 9903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03aac2d3dc9a4b478f0cf7459f7e63db [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:56:02.420641 10009 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03aac2d3dc9a4b478f0cf7459f7e63db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:56:02.420699 10009 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:56:02.419374 9835 master_runner.cc:186] Error getting master registration for 127.19.34.126:43065: OK, instance_id { permanent_uuid: "bc103509eebc4deea2e6b182a14a8196" instance_seqno: 1781880961972015 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:56:02.422135 9835 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:56:02.431005 9828 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } attempt: 3
Jun 19 14:56:02 dist-test-slave-3qdh krb5kdc[9747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880961, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.34.126@KRBTEST.COM for kudu/127.19.34.124@KRBTEST.COM
I20260619 14:56:02.439147 9832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.431366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.34.124:43245 (local address 127.0.0.1:35022)
0619 14:56:02.431626 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 14:56:02.431639 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:56:02.431716 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:56:02.431926 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:56:02.431929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:56:02.432134 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:56:02.432226 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:56:02.432232 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.432855 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.432860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:56:02.433631 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:56:02.433639 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.434012 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.434015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:56:02.434103 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.434613 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:56:02.434629 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:56:02.436072 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:56:02.437839 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:56:02.437842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:56:02.437844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:56:02.438027 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:56:02.438285 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:56:02.438288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:56:02.438290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:56:02.438364 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:56:02.438719 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:56:02.438722 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:56:02.438803 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 14:56:02.438892 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 14:56:02.439000 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198}
I20260619 14:56:02.439147 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.431432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.19.34.124:43245)
0619 14:56:02.431546 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.431550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.431689 (+ 139us) server_negotiation.cc:407] Connection header received
0619 14:56:02.431789 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.431793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.431835 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.431907 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.432328 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.432759 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.433718 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.433914 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.434010 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.436211 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:56:02.436215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:56:02.436218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:56:02.436242 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:56:02.437746 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.438135 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.438139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.438144 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.438215 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.438446 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.438452 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.438453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.438641 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:56:02.438718 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:02.438944 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 14:56:02.439030 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:56:02.441712 9828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196: Bootstrap starting.
I20260619 14:56:02.443528 9828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196: Neither blocks nor log segments found. Creating new log.
I20260619 14:56:02.444172 9828 log.cc:824] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196: Log is configured to *not* fsync() on all Append() calls
I20260619 14:56:02.446009 9828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196: No bootstrap required, opened a new log
I20260619 14:56:02.448638 9828 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc103509eebc4deea2e6b182a14a8196" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43065 } } peers { permanent_uuid: "03aac2d3dc9a4b478f0cf7459f7e63db" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } } peers { permanent_uuid: "6e22926006444dd7ba9e5cac38cc1739" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } }
I20260619 14:56:02.448822 9828 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc103509eebc4deea2e6b182a14a8196, State: Initialized, Role: FOLLOWER
I20260619 14:56:02.449115 9828 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc103509eebc4deea2e6b182a14a8196" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43065 } } peers { permanent_uuid: "03aac2d3dc9a4b478f0cf7459f7e63db" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } } peers { permanent_uuid: "6e22926006444dd7ba9e5cac38cc1739" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } }
I20260619 14:56:02.449491 10012 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc103509eebc4deea2e6b182a14a8196" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43065 } } peers { permanent_uuid: "03aac2d3dc9a4b478f0cf7459f7e63db" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } } peers { permanent_uuid: "6e22926006444dd7ba9e5cac38cc1739" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } } }
I20260619 14:56:02.449683 10012 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:56:02.450174 9828 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:56:02.450253 9802 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 14:56:02.450326 9802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:56:02.451390 9802 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc103509eebc4deea2e6b182a14a8196" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43065 } } peers { permanent_uuid: "03aac2d3dc9a4b478f0cf7459f7e63db" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } } peers { permanent_uuid: "6e22926006444dd7ba9e5cac38cc1739" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } }
I20260619 14:56:02.452225 9802 leader_election.cc:288] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [CANDIDATE]: Term 1 election: Requested vote from peers 03aac2d3dc9a4b478f0cf7459f7e63db (127.19.34.125:43051), 6e22926006444dd7ba9e5cac38cc1739 (127.19.34.124:43245)
I20260619 14:56:02.452862 9952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc103509eebc4deea2e6b182a14a8196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6e22926006444dd7ba9e5cac38cc1739"
I20260619 14:56:02.453064 9952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e22926006444dd7ba9e5cac38cc1739 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:56:02.454205 9952 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6e22926006444dd7ba9e5cac38cc1739 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc103509eebc4deea2e6b182a14a8196 in term 1.
I20260619 14:56:02.454959 9877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc103509eebc4deea2e6b182a14a8196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "03aac2d3dc9a4b478f0cf7459f7e63db"
I20260619 14:56:02.455160 9877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03aac2d3dc9a4b478f0cf7459f7e63db [term 0 FOLLOWER]: Advancing to term 1
W20260619 14:56:02.455322 10023 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:56:02.455376 10023 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:56:02.456218 9877 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 03aac2d3dc9a4b478f0cf7459f7e63db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc103509eebc4deea2e6b182a14a8196 in term 1.
I20260619 14:56:02.456625 9779 leader_election.cc:302] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03aac2d3dc9a4b478f0cf7459f7e63db, bc103509eebc4deea2e6b182a14a8196; no voters:
I20260619 14:56:02.456840 10012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:56:02.456977 10012 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [term 1 LEADER]: Becoming Leader. State: Replica: bc103509eebc4deea2e6b182a14a8196, State: Running, Role: LEADER
I20260619 14:56:02.457283 10012 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc103509eebc4deea2e6b182a14a8196" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43065 } } peers { permanent_uuid: "03aac2d3dc9a4b478f0cf7459f7e63db" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } } peers { permanent_uuid: "6e22926006444dd7ba9e5cac38cc1739" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } }
I20260619 14:56:02.459391 10012 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc103509eebc4deea2e6b182a14a8196. Latest consensus state: current_term: 1 leader_uuid: "bc103509eebc4deea2e6b182a14a8196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc103509eebc4deea2e6b182a14a8196" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43065 } } peers { permanent_uuid: "03aac2d3dc9a4b478f0cf7459f7e63db" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } } peers { permanent_uuid: "6e22926006444dd7ba9e5cac38cc1739" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } } }
I20260619 14:56:02.459481 10012 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [sys.catalog]: This master's current role is: LEADER
I20260619 14:56:02.461743 10031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:56:02.462533 10031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 14:56:02.468564 9781 tablet.cc:2369] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:56:02.474006 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.464792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35036 (local address 127.19.34.124:43245)
0619 14:56:02.464881 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.464887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.465630 (+ 743us) server_negotiation.cc:407] Connection header received
0619 14:56:02.465821 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.465826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.465875 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.465925 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.466920 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.467622 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.468746 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.469016 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.469078 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.470654 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:56:02.470657 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:56:02.470660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:56:02.470679 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:56:02.472327 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.472760 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.472763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.472765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.472834 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.473074 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.473078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.473079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.473279 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:56:02.473370 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:02.473767 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 14:56:02.473864 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260619 14:56:02.477900 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.465137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53760 (local address 127.19.34.126:43065)
0619 14:56:02.465257 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.465261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.466653 (+ 1392us) server_negotiation.cc:407] Connection header received
0619 14:56:02.466806 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.466808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.466852 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.466901 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.467689 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.468400 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.470491 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.470761 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.470832 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.474612 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:56:02.474616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:56:02.474618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:56:02.474640 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:56:02.476031 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.476580 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.476585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.476587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.476671 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.476977 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.476981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.476984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.477180 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:56:02.477286 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:02.477612 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 14:56:02.477731 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:56:02.482375 9952 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6e22926006444dd7ba9e5cac38cc1739 [term 1 FOLLOWER]: Refusing update from remote peer bc103509eebc4deea2e6b182a14a8196: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:56:02.483136 9877 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03aac2d3dc9a4b478f0cf7459f7e63db [term 1 FOLLOWER]: Refusing update from remote peer bc103509eebc4deea2e6b182a14a8196: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:56:02.483868 10012 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03aac2d3dc9a4b478f0cf7459f7e63db" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:56:02.484263 10012 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e22926006444dd7ba9e5cac38cc1739" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:56:02.487948 9998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03aac2d3dc9a4b478f0cf7459f7e63db [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc103509eebc4deea2e6b182a14a8196. Latest consensus state: current_term: 1 leader_uuid: "bc103509eebc4deea2e6b182a14a8196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc103509eebc4deea2e6b182a14a8196" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43065 } } peers { permanent_uuid: "03aac2d3dc9a4b478f0cf7459f7e63db" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } } peers { permanent_uuid: "6e22926006444dd7ba9e5cac38cc1739" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } } }
I20260619 14:56:02.488060 9998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03aac2d3dc9a4b478f0cf7459f7e63db [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:56:02.491544 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:02.465908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.19.34.125:43051)
0619 14:56:02.466023 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.466029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.468726 (+ 2697us) server_negotiation.cc:407] Connection header received
0619 14:56:02.469641 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.469645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.469688 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.469741 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.471285 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.471998 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.473931 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:02.474240 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:02.474305 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:02.477357 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:56:02.477360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:56:02.477363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:56:02.477385 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:56:02.478849 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.479557 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.479560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.479562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.479661 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:56:02.479913 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:56:02.479916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:56:02.479917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:56:02.480080 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:56:02.480198 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:02.491239 (+ 11041us) server_negotiation.cc:299] Negotiation successful
0619 14:56:02.491401 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:56:02.491852 9984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e22926006444dd7ba9e5cac38cc1739 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc103509eebc4deea2e6b182a14a8196. Latest consensus state: current_term: 1 leader_uuid: "bc103509eebc4deea2e6b182a14a8196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc103509eebc4deea2e6b182a14a8196" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43065 } } peers { permanent_uuid: "03aac2d3dc9a4b478f0cf7459f7e63db" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } } peers { permanent_uuid: "6e22926006444dd7ba9e5cac38cc1739" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } } }
I20260619 14:56:02.491966 9984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e22926006444dd7ba9e5cac38cc1739 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:56:02.494644 10025 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc103509eebc4deea2e6b182a14a8196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc103509eebc4deea2e6b182a14a8196" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43065 } } peers { permanent_uuid: "03aac2d3dc9a4b478f0cf7459f7e63db" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } } peers { permanent_uuid: "6e22926006444dd7ba9e5cac38cc1739" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } } }
I20260619 14:56:02.494750 10025 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [sys.catalog]: This master's current role is: LEADER
I20260619 14:56:02.494864 10032 mvcc.cc:205] Tried to move back new op lower bound from 7298584422304321536 to 7298584422227881984. Current Snapshot: MvccSnapshot[applied={T|T < 7298584422304321536}]
I20260619 14:56:02.495718 9998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03aac2d3dc9a4b478f0cf7459f7e63db [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc103509eebc4deea2e6b182a14a8196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc103509eebc4deea2e6b182a14a8196" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43065 } } peers { permanent_uuid: "03aac2d3dc9a4b478f0cf7459f7e63db" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } } peers { permanent_uuid: "6e22926006444dd7ba9e5cac38cc1739" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } } }
I20260619 14:56:02.495823 9998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03aac2d3dc9a4b478f0cf7459f7e63db [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:56:02.498026 10036 mvcc.cc:205] Tried to move back new op lower bound from 7298584422304321536 to 7298584422227881984. Current Snapshot: MvccSnapshot[applied={T|T < 7298584422304321536}]
I20260619 14:56:02.499429 10031 catalog_manager.cc:1347] Generated new cluster ID: 047c8e55272e438f853b08f4e47d719b
I20260619 14:56:02.499482 10031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:56:02.505865 10025 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc103509eebc4deea2e6b182a14a8196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc103509eebc4deea2e6b182a14a8196" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43065 } } peers { permanent_uuid: "03aac2d3dc9a4b478f0cf7459f7e63db" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } } peers { permanent_uuid: "6e22926006444dd7ba9e5cac38cc1739" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } } }
I20260619 14:56:02.505967 10025 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196 [sys.catalog]: This master's current role is: LEADER
I20260619 14:56:02.507019 9984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e22926006444dd7ba9e5cac38cc1739 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc103509eebc4deea2e6b182a14a8196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc103509eebc4deea2e6b182a14a8196" member_type: VOTER last_known_addr { host: "127.19.34.126" port: 43065 } } peers { permanent_uuid: "03aac2d3dc9a4b478f0cf7459f7e63db" member_type: VOTER last_known_addr { host: "127.19.34.125" port: 43051 } } peers { permanent_uuid: "6e22926006444dd7ba9e5cac38cc1739" member_type: VOTER last_known_addr { host: "127.19.34.124" port: 43245 } } }
I20260619 14:56:02.507120 9984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e22926006444dd7ba9e5cac38cc1739 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:56:02.521068 10031 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:56:02.522320 10031 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:56:02.544379 10031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc103509eebc4deea2e6b182a14a8196: Generated new TSK 0
I20260619 14:56:02.545135 10031 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:56:02.571686 9831 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:56:02.568719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53764 (local address 127.19.34.126:43065)
0619 14:56:02.568867 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.568873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.570689 (+ 1816us) server_negotiation.cc:407] Connection header received
0619 14:56:02.570872 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.570877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.570927 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.570977 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.571542 (+ 565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53764: BlockingRecv error: recv got EOF from 127.0.0.1:53764 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 14:56:02.572996 9908 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:56:02.569487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.19.34.125:43051)
0619 14:56:02.569614 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.569620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.572066 (+ 2446us) server_negotiation.cc:407] Connection header received
0619 14:56:02.572255 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.572259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.572305 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.572375 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.572893 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59416: BlockingRecv error: recv got EOF from 127.0.0.1:59416 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 14:56:02.574496 9981 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:56:02.570149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35052 (local address 127.19.34.124:43245)
0619 14:56:02.570251 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.570256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.573413 (+ 3157us) server_negotiation.cc:407] Connection header received
0619 14:56:02.573777 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.573781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.573835 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.573890 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.574378 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35052: BlockingRecv error: recv got EOF from 127.0.0.1:35052 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260619 14:56:02.578886 9908 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:56:02.577327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59428 (local address 127.19.34.125:43051)
0619 14:56:02.577425 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.577428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.577950 (+ 522us) server_negotiation.cc:407] Connection header received
0619 14:56:02.578160 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.578163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.578237 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.578295 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.578798 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59428: BlockingRecv error: recv got EOF from 127.0.0.1:59428 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 14:56:02.583559 9908 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:56:02.582114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.19.34.125:43051)
0619 14:56:02.582212 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.582216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.582730 (+ 514us) server_negotiation.cc:407] Connection header received
0619 14:56:02.582926 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.582929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.582975 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.583024 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.583479 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59434: BlockingRecv error: recv got EOF from 127.0.0.1:59434 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260619 14:56:02.588940 9908 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:56:02.587320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59438 (local address 127.19.34.125:43051)
0619 14:56:02.587404 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.587408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.587937 (+ 529us) server_negotiation.cc:407] Connection header received
0619 14:56:02.588115 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.588118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.588164 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.588218 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.588841 (+ 623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59438: BlockingRecv error: recv got EOF from 127.0.0.1:59438 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260619 14:56:02.595850 9908 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:56:02.594557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59448 (local address 127.19.34.125:43051)
0619 14:56:02.594672 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.594675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.595098 (+ 423us) server_negotiation.cc:407] Connection header received
0619 14:56:02.595265 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.595268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.595308 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.595352 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.595770 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59448: BlockingRecv error: recv got EOF from 127.0.0.1:59448 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 14:56:02.606715 9908 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:56:02.605686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59460 (local address 127.19.34.125:43051)
0619 14:56:02.605776 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.605780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.606025 (+ 245us) server_negotiation.cc:407] Connection header received
0619 14:56:02.606109 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.606113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.606164 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.606248 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.606638 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59460: BlockingRecv error: recv got EOF from 127.0.0.1:59460 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260619 14:56:02.625532 9908 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:56:02.624346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59472 (local address 127.19.34.125:43051)
0619 14:56:02.624470 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.624474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.624687 (+ 213us) server_negotiation.cc:407] Connection header received
0619 14:56:02.624808 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.624810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.624848 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.624907 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.625429 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59472: BlockingRecv error: recv got EOF from 127.0.0.1:59472 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 14:56:02.660463 9908 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:56:02.659540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.19.34.125:43051)
0619 14:56:02.659682 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.659685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.659749 (+ 64us) server_negotiation.cc:407] Connection header received
0619 14:56:02.659850 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.659852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.659893 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.659954 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.660340 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59486: BlockingRecv error: recv got EOF from 127.0.0.1:59486 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 14:56:02.727669 9908 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:56:02.726343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59494 (local address 127.19.34.125:43051)
0619 14:56:02.726554 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.726558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.726730 (+ 172us) server_negotiation.cc:407] Connection header received
0619 14:56:02.726940 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.726943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.726992 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.727078 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.727561 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59494: BlockingRecv error: recv got EOF from 127.0.0.1:59494 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260619 14:56:02.858798 9908 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:56:02.857726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59506 (local address 127.19.34.125:43051)
0619 14:56:02.857871 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:02.857875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:02.858037 (+ 162us) server_negotiation.cc:407] Connection header received
0619 14:56:02.858138 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:02.858140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:02.858186 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:02.858251 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:02.858679 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59506: BlockingRecv error: recv got EOF from 127.0.0.1:59506 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 14:56:03.117805 9908 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:56:03.116718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59518 (local address 127.19.34.125:43051)
0619 14:56:03.116836 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:03.116840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:03.117004 (+ 164us) server_negotiation.cc:407] Connection header received
0619 14:56:03.117133 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:03.117136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:03.117193 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:03.117258 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:56:03.117677 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59518: BlockingRecv error: recv got EOF from 127.0.0.1:59518 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:56:03.393484 9995 catalog_manager.cc:1259] Loaded cluster ID: 047c8e55272e438f853b08f4e47d719b
I20260619 14:56:03.393553 9995 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6e22926006444dd7ba9e5cac38cc1739: loading cluster ID for follower catalog manager: success
I20260619 14:56:03.394958 9995 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6e22926006444dd7ba9e5cac38cc1739: acquiring CA information for follower catalog manager: success
I20260619 14:56:03.395999 9995 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6e22926006444dd7ba9e5cac38cc1739: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:56:03.421756 10009 catalog_manager.cc:1259] Loaded cluster ID: 047c8e55272e438f853b08f4e47d719b
I20260619 14:56:03.421813 10009 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 03aac2d3dc9a4b478f0cf7459f7e63db: loading cluster ID for follower catalog manager: success
I20260619 14:56:03.423282 10009 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 03aac2d3dc9a4b478f0cf7459f7e63db: acquiring CA information for follower catalog manager: success
I20260619 14:56:03.424265 10009 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 03aac2d3dc9a4b478f0cf7459f7e63db: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:56:03.634840 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:03.631576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.19.34.125:43051)
0619 14:56:03.631813 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:03.631816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:03.631881 (+ 65us) server_negotiation.cc:407] Connection header received
0619 14:56:03.631999 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:03.632001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:03.632124 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:03.632209 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:56:03.632416 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:03.632870 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:03.633555 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:03.633743 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:03.633817 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:03.634036 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:56:03.634469 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:56:03.634553 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:03.634599 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 14:56:03.634642 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":62,"threads_started":1}
I20260619 14:56:03.640493 10113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:03.637596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.19.34.124:43245)
0619 14:56:03.637783 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:03.637786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:03.637798 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:56:03.637902 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:03.637904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:03.638001 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:03.638081 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:56:03.638229 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:03.638653 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:03.639416 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:03.639595 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:03.639658 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:03.639876 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:56:03.640173 (+ 297us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:56:03.640244 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:03.640345 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 14:56:03.640397 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
I20260619 14:56:03.647064 10113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:03.643178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35068 (local address 127.19.34.124:43245)
0619 14:56:03.643272 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:03.643276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:03.643559 (+ 283us) server_negotiation.cc:407] Connection header received
0619 14:56:03.643678 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:03.643681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:03.643717 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:03.643775 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:56:03.644053 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:03.644736 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:03.645639 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:03.645969 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:03.646077 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:03.646313 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:56:03.646382 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:56:03.646460 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:03.646906 (+ 446us) server_negotiation.cc:299] Negotiation successful
0619 14:56:03.646953 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260619 14:56:03.647480 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:03.643048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53774 (local address 127.19.34.126:43065)
0619 14:56:03.643330 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:03.643334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:03.643391 (+ 57us) server_negotiation.cc:407] Connection header received
0619 14:56:03.643566 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:03.643569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:03.643695 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:03.644004 (+ 309us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:56:03.644129 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:03.644820 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:03.646023 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:03.646376 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:03.646536 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:03.646796 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:56:03.647038 (+ 242us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:56:03.647252 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:03.647285 (+ 33us) server_negotiation.cc:299] Negotiation successful
0619 14:56:03.647342 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":81,"threads_started":1}
I20260619 14:56:03.647606 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:56:03.643300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59526 (local address 127.19.34.125:43051)
0619 14:56:03.643399 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0619 14:56:03.643403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:56:03.643784 (+ 381us) server_negotiation.cc:407] Connection header received
0619 14:56:03.643912 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:56:03.643915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:56:03.643958 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:56:03.644040 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:56:03.644752 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:03.645458 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:03.646466 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:56:03.646789 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:56:03.646886 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:56:03.647109 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:56:03.647178 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:56:03.647232 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 14:56:03.647429 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 14:56:03.647486 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:56:03.654778 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 9760
I20260619 14:56:03.660784 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 9835
I20260619 14:56:03.666152 19593 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0Qcekt/build/debug/bin/kudu with pid 9910
2026-06-19T14:56:03Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2923 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5843 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (266644 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST